[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:21 dist-test-slave-smns krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:21 dist-test-slave-smns krb5kdc[399](info): set up 2 sockets
Sep 28 09:04:21 dist-test-slave-smns krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 09:04:22.605957 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:22 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:22.729372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:22.810254 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:22.810489 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:22.810524 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:22.811679 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:22.811743 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:22.811767 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:22.811789 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:22.811813 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:22.813364 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:22.813699 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:22.813956 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:04:22.819528 412 server_base.cc:1034] running on GCE node
W20260928 09:04:22.819573 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:22.819550 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:22.819550 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:22.819949 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:22.820255 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:22.821477 412 hybrid_clock.cc:648] HybridClock initialized: now 1790586262821453 us; error 38 us; skew 500 ppm
Sep 28 09:04:22 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:22.823972 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:22.824474 412 webserver.cc:460] Webserver started at http://127.0.98.62:33479/ using document root <none> and password file <none>
I20260928 09:04:22.824669 412 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:22.824708 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:22.824802 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:22.825918 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "9932480cf2c14506805637ce5f625a50"
format_stamp: "Formatted at 2026-09-28 09:04:22 on dist-test-slave-smns"
server_key: "8c13b95a516596fead79830b91c0505b"
server_key_iv: "091be2032f6ba362cfd1e1869f77e87d"
server_key_version: "encryptionkey@0"
I20260928 09:04:22.826239 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "9932480cf2c14506805637ce5f625a50"
format_stamp: "Formatted at 2026-09-28 09:04:22 on dist-test-slave-smns"
server_key: "8c13b95a516596fead79830b91c0505b"
server_key_iv: "091be2032f6ba362cfd1e1869f77e87d"
server_key_version: "encryptionkey@0"
I20260928 09:04:22.827548 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:22.828229 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:22.828426 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:22.828506 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "9932480cf2c14506805637ce5f625a50"
format_stamp: "Formatted at 2026-09-28 09:04:22 on dist-test-slave-smns"
server_key: "8c13b95a516596fead79830b91c0505b"
server_key_iv: "091be2032f6ba362cfd1e1869f77e87d"
server_key_version: "encryptionkey@0"
I20260928 09:04:22.828560 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:22.840435 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:22.843186 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:22.843374 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:22.848044 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33905
I20260928 09:04:22.848105 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33905 every 8 connection(s)
I20260928 09:04:22.848428 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:22.849339 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:22.851573 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50: Bootstrap starting.
I20260928 09:04:22.852083 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:22.852304 480 log.cc:824] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:22.852949 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50: No bootstrap required, opened a new log
I20260928 09:04:22.853914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 412
I20260928 09:04:22.854001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:22.854622 392 external_mini_cluster.cc:1428] Setting key a63993707b4fbcd48753a921bbea7a71
I20260928 09:04:22.854679 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9932480cf2c14506805637ce5f625a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260928 09:04:22.854852 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:22.854898 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9932480cf2c14506805637ce5f625a50, State: Initialized, Role: FOLLOWER
I20260928 09:04:22.855078 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9932480cf2c14506805637ce5f625a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260928 09:04:22.855192 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:22.855238 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:22.855280 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:22.856259 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9932480cf2c14506805637ce5f625a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260928 09:04:22.856387 480 leader_election.cc:302] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9932480cf2c14506805637ce5f625a50; no voters:
I20260928 09:04:22.856575 480 leader_election.cc:288] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:22.856594 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:22.856747 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [term 1 LEADER]: Becoming Leader. State: Replica: 9932480cf2c14506805637ce5f625a50, State: Running, Role: LEADER
I20260928 09:04:22.856873 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:22.856930 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9932480cf2c14506805637ce5f625a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260928 09:04:22.857286 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9932480cf2c14506805637ce5f625a50. Latest consensus state: current_term: 1 leader_uuid: "9932480cf2c14506805637ce5f625a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9932480cf2c14506805637ce5f625a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } } }
I20260928 09:04:22.857292 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9932480cf2c14506805637ce5f625a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9932480cf2c14506805637ce5f625a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } } }
I20260928 09:04:22.857466 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:22.857478 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:22.857746 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:22.858345 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:22.859894 494 catalog_manager.cc:1347] Generated new cluster ID: b64c604a87df4e5493fe35e1c107433f
I20260928 09:04:22.859935 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:04:22 dist-test-slave-smns krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:22.871981 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:22.855495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.0.98.62:33905)
0928 09:04:22.855670 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:22.855679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:22.855932 (+ 253us) server_negotiation.cc:407] Connection header received
0928 09:04:22.858195 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:22.858203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:22.858259 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:22.858381 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:22.860443 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:22.861553 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:22.862982 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:22.863634 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:22.863750 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:22.867969 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:22.867973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:22.867973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:22.867983 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:22.869980 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:22.870488 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:22.870490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:22.870491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:22.870591 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:22.870886 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:22.870887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:22.870887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:22.871167 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:22.871323 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:22.871622 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 09:04:22.871750 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260928 09:04:22.878044 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:22.878577 494 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:22.888201 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9932480cf2c14506805637ce5f625a50: Generated new TSK 0
I20260928 09:04:22.888386 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:22.920876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:23.005739 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:23.005965 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:23.005991 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:23.007175 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:23.007220 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:23.007272 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:23.008716 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:23.008975 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:23.009212 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:23.014226 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:23.014235 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:23.014235 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:23.014283 506 server_base.cc:1034] running on GCE node
I20260928 09:04:23.014622 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:23.014859 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:23.016023 506 hybrid_clock.cc:648] HybridClock initialized: now 1790586263016005 us; error 36 us; skew 500 ppm
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:23.018463 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:23.018879 506 webserver.cc:460] Webserver started at http://127.0.98.1:33737/ using document root <none> and password file <none>
I20260928 09:04:23.019097 506 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:23.019164 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:23.019316 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:23.020368 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "a989458a11d542dcba693815db878291"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "17e7e4a6de0877b9c1859e5aa0568ca6"
server_key_iv: "e7867a8f64f2b3c26ae2e5062234415b"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.020756 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a989458a11d542dcba693815db878291"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "17e7e4a6de0877b9c1859e5aa0568ca6"
server_key_iv: "e7867a8f64f2b3c26ae2e5062234415b"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.022163 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:23.022846 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.023005 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:23.023101 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "a989458a11d542dcba693815db878291"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "17e7e4a6de0877b9c1859e5aa0568ca6"
server_key_iv: "e7867a8f64f2b3c26ae2e5062234415b"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.023180 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:23.051607 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:23.053687 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:23.053875 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:23.054128 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:23.054440 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:23.054493 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.054527 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:23.054575 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.060554 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45081
I20260928 09:04:23.060634 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45081 every 8 connection(s)
I20260928 09:04:23.060971 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:23.065928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 506
I20260928 09:04:23.066004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:23.066166 392 external_mini_cluster.cc:1428] Setting key 3dcdce8cf4225d93ebafb4708a7ca68c
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:23.071202 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.061727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33905 (local address 127.0.98.1:42885)
0928 09:04:23.061875 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:23.061915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:23.062473 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:23.062707 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:23.062710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:23.063292 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:23.063486 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:23.063490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.064212 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.064213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:23.065042 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:23.065044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.065464 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.065465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:23.065663 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.066256 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:23.066272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:23.067824 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:23.069847 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:23.069853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:23.069855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:23.070079 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:23.070309 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:23.070310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:23.070310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:23.070428 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:23.070720 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:23.070724 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:23.070838 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 09:04:23.070922 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:04:23.071040 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 09:04:23.071208 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.061751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42885 (local address 127.0.98.62:33905)
0928 09:04:23.061795 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:23.061797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:23.062005 (+ 208us) server_negotiation.cc:407] Connection header received
0928 09:04:23.062578 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:23.062580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:23.062592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:23.062640 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:23.063596 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.064115 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.065143 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.065364 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.065452 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.067975 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:23.067978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:23.067978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:23.067987 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:23.069763 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.070161 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.070162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.070163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.070229 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.070499 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.070500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.070501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.070653 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:23.070727 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:23.070972 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 09:04:23.071073 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:04:23.071853 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33905
I20260928 09:04:23.071949 635 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:23.072137 635 heartbeater.cc:511] Master 127.0.98.62:33905 requested a full tablet report, sending...
I20260928 09:04:23.072695 444 ts_manager.cc:194] Registered new tserver with Master: a989458a11d542dcba693815db878291 (127.0.98.1:45081)
I20260928 09:04:23.073297 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:23.094120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:23.177980 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:23.178303 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:23.178345 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:23.179533 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:23.179576 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:23.179603 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:23.181063 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:23.181303 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:23.181587 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:04:23.186731 640 server_base.cc:1034] running on GCE node
W20260928 09:04:23.186739 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:23.186748 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:23.186959 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:23.187254 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:23.187673 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:23.188851 640 hybrid_clock.cc:648] HybridClock initialized: now 1790586263188836 us; error 27 us; skew 500 ppm
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:23.191475 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:23.191917 640 webserver.cc:460] Webserver started at http://127.0.98.2:46787/ using document root <none> and password file <none>
I20260928 09:04:23.192139 640 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:23.192183 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:23.192332 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:23.193394 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "228501b0d827417ab17d8b55108e1f2e"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "93f9daea516f7345cad90d5a318b4ad3"
server_key_iv: "bbf148715bab8f38dbb77b688cc0f931"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.193812 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "228501b0d827417ab17d8b55108e1f2e"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "93f9daea516f7345cad90d5a318b4ad3"
server_key_iv: "bbf148715bab8f38dbb77b688cc0f931"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.195190 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:23.195919 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.196097 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:23.196192 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "228501b0d827417ab17d8b55108e1f2e"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "93f9daea516f7345cad90d5a318b4ad3"
server_key_iv: "bbf148715bab8f38dbb77b688cc0f931"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.196247 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:23.203740 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:23.205642 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:23.205755 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:23.205926 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:23.206172 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:23.206202 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.206221 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:23.206233 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.212881 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37501
I20260928 09:04:23.213310 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:23.214835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 640
I20260928 09:04:23.214947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:23.215107 392 external_mini_cluster.cc:1428] Setting key b9d3f0c07b45596fe0f327701ba160f9
I20260928 09:04:23.214287 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37501 every 8 connection(s)
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:04:23.225646 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.216011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33077 (local address 127.0.98.62:33905)
0928 09:04:23.216144 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:23.216147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:23.216351 (+ 204us) server_negotiation.cc:407] Connection header received
0928 09:04:23.217089 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:23.217090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:23.217103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:23.217179 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:23.218176 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.218683 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.219767 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.219986 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.220078 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.222611 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:23.222612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:23.222613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:23.222620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:23.224094 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.224597 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.224599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.224599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.224663 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.224912 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.224913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.224913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.225028 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:23.225100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:23.225390 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 09:04:23.225490 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 09:04:23.225785 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.216035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33905 (local address 127.0.98.2:33077)
0928 09:04:23.216212 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:23.216253 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:23.216979 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:23.217233 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:23.217237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:23.217882 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:23.218072 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:23.218077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.218777 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.218778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:23.219667 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:23.219670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.220090 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.220091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:23.220210 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.220767 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:23.220775 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:23.222500 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:23.224194 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:23.224200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:23.224202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:23.224509 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:23.224729 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:23.224730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:23.224731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:23.224844 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:23.225105 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:23.225111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:23.225246 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 09:04:23.225356 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 09:04:23.225510 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 09:04:23.226337 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33905
I20260928 09:04:23.226436 769 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:23.226605 769 heartbeater.cc:511] Master 127.0.98.62:33905 requested a full tablet report, sending...
I20260928 09:04:23.227134 444 ts_manager.cc:194] Registered new tserver with Master: 228501b0d827417ab17d8b55108e1f2e (127.0.98.2:37501)
I20260928 09:04:23.227517 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:23.242803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:23.328467 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:23.328671 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:23.328696 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:23.329864 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:23.329903 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:23.329931 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:23.331452 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:23.331637 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:23.331828 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:23.336894 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:23.337121 774 server_base.cc:1034] running on GCE node
W20260928 09:04:23.336910 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:23.336931 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:23.337519 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:23.337793 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:23.338960 774 hybrid_clock.cc:648] HybridClock initialized: now 1790586263338962 us; error 41 us; skew 500 ppm
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:23.341837 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:23.342285 774 webserver.cc:460] Webserver started at http://127.0.98.3:33769/ using document root <none> and password file <none>
I20260928 09:04:23.342525 774 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:23.342592 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:23.342736 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:23.343811 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed5343ce7b0f484383b85c8f3bbc48b8"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "4f83f450693dbf0571221ffbec523bbd"
server_key_iv: "3910ecf1194a9b8ff214de9a7e9792de"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.344170 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed5343ce7b0f484383b85c8f3bbc48b8"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "4f83f450693dbf0571221ffbec523bbd"
server_key_iv: "3910ecf1194a9b8ff214de9a7e9792de"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.345696 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:23.346465 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.346706 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:23.346827 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "ed5343ce7b0f484383b85c8f3bbc48b8"
format_stamp: "Formatted at 2026-09-28 09:04:23 on dist-test-slave-smns"
server_key: "4f83f450693dbf0571221ffbec523bbd"
server_key_iv: "3910ecf1194a9b8ff214de9a7e9792de"
server_key_version: "encryptionkey@0"
I20260928 09:04:23.346913 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:23.357375 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:23.359730 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:23.359839 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:23.360074 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:23.360363 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:23.360400 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.360443 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:23.360474 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:23.366832 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35165
I20260928 09:04:23.366954 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35165 every 8 connection(s)
I20260928 09:04:23.367228 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:23.367821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 774
I20260928 09:04:23.367926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:23.368101 392 external_mini_cluster.cc:1428] Setting key 65a9de7a4317952f5b0835d1c6781197
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:23.376981 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.367961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50815 (local address 127.0.98.62:33905)
0928 09:04:23.368016 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:23.368019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:23.368213 (+ 194us) server_negotiation.cc:407] Connection header received
0928 09:04:23.368862 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:23.368863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:23.368873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:23.368941 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:23.369991 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.370489 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.371477 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.371662 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.371721 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.374332 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:23.374333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:23.374334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:23.374340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:23.375663 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.376128 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.376129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.376129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.376176 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.376437 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.376438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.376438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.376545 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:23.376601 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:23.376844 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 09:04:23.376914 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:04:23.377154 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.367936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33905 (local address 127.0.98.3:50815)
0928 09:04:23.368080 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:23.368122 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:23.368766 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:23.369012 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:23.369016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:23.369708 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:23.369902 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:23.369907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.370576 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.370576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:23.371389 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:23.371391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.371735 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.371735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:23.371826 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.372372 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:23.372385 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:23.374247 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:23.375818 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:23.375824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:23.375826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:23.376044 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:23.376253 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:23.376255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:23.376255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:23.376369 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:23.376604 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:23.376609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:23.376723 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 09:04:23.376814 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 09:04:23.376934 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260928 09:04:23.377707 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33905
I20260928 09:04:23.377815 903 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:23.378000 903 heartbeater.cc:511] Master 127.0.98.62:33905 requested a full tablet report, sending...
I20260928 09:04:23.378405 444 ts_manager.cc:194] Registered new tserver with Master: ed5343ce7b0f484383b85c8f3bbc48b8 (127.0.98.3:35165)
I20260928 09:04:23.378786 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50815
I20260928 09:04:23.380062 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:23.402650 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.394807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.98.1:45081)
0928 09:04:23.394959 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:23.394964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:23.394974 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:23.395000 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:23.395002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:23.395016 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:23.395081 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:23.395641 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.396383 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.397143 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.397467 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.397521 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.399925 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:23.399934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:23.399934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:23.399941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:23.401679 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.401988 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.401991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.401992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.402043 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.402204 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.402205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.402205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.402328 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:23.402444 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:23.402509 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:04:23.402571 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260928 09:04:23.403054 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37462
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:23.425038 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:23.418232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.0.98.1:45081)
0928 09:04:23.418300 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:23.418302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:23.418352 (+ 50us) server_negotiation.cc:407] Connection header received
0928 09:04:23.418410 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:23.418411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:23.418422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:23.418467 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:23.419037 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.419555 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.420305 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:23.420503 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:23.420559 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:23.422741 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:23.422742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:23.422742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:23.422749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:23.424135 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.424421 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.424422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.424422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.424470 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:23.424614 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:23.424615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:23.424615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:23.424693 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:23.424774 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:23.424839 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:04:23.424926 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:04:23.425709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 506
I20260928 09:04:23.431391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 640
I20260928 09:04:23.436791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 774
I20260928 09:04:23.441057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 412
2026-09-28T09:04:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1881 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[917](info): set up 2 sockets
Sep 28 09:04:23 dist-test-slave-smns krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 09:04:24.469060 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:24 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:24.589383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35649
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:24.672617 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:24.672866 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:24.672902 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:24.674072 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:24.674141 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:24.674166 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:24.674188 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:24.674216 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:24.675648 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:24.675942 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:24.676180 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:24.681080 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:24.681155 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:24.681155 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:24.681238 930 server_base.cc:1034] running on GCE node
I20260928 09:04:24.681584 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:24.681919 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:24.683064 930 hybrid_clock.cc:648] HybridClock initialized: now 1790586264683043 us; error 39 us; skew 500 ppm
Sep 28 09:04:24 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:24.690933 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:24.691399 930 webserver.cc:460] Webserver started at http://127.0.98.62:44389/ using document root <none> and password file <none>
I20260928 09:04:24.691641 930 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:24.691690 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:24.691870 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:24.692986 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0956e84ae01d4f0c98c9227ac4b15ff2"
format_stamp: "Formatted at 2026-09-28 09:04:24 on dist-test-slave-smns"
server_key: "3d06926a62a9fce2ff0fd5fa2abdd60b"
server_key_iv: "f81f11f16f607bb9ec292b8b01f13f03"
server_key_version: "encryptionkey@0"
I20260928 09:04:24.693393 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0956e84ae01d4f0c98c9227ac4b15ff2"
format_stamp: "Formatted at 2026-09-28 09:04:24 on dist-test-slave-smns"
server_key: "3d06926a62a9fce2ff0fd5fa2abdd60b"
server_key_iv: "f81f11f16f607bb9ec292b8b01f13f03"
server_key_version: "encryptionkey@0"
I20260928 09:04:24.694774 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:24.695559 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:24.695757 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:24.695837 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0956e84ae01d4f0c98c9227ac4b15ff2"
format_stamp: "Formatted at 2026-09-28 09:04:24 on dist-test-slave-smns"
server_key: "3d06926a62a9fce2ff0fd5fa2abdd60b"
server_key_iv: "f81f11f16f607bb9ec292b8b01f13f03"
server_key_version: "encryptionkey@0"
I20260928 09:04:24.695915 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:24.704329 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:24.706403 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:24.706593 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:24.710513 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35649
I20260928 09:04:24.710559 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35649 every 8 connection(s)
I20260928 09:04:24.710914 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:24.711831 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:24.714038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 930
I20260928 09:04:24.714134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:24.714291 392 external_mini_cluster.cc:1428] Setting key 172cb8404883d6c8d525ffd00097fc21
I20260928 09:04:24.714362 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2: Bootstrap starting.
I20260928 09:04:24.715019 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:24.715313 998 log.cc:824] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:24.716063 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2: No bootstrap required, opened a new log
I20260928 09:04:24.717940 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35649 } }
I20260928 09:04:24.718094 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:24.718138 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0956e84ae01d4f0c98c9227ac4b15ff2, State: Initialized, Role: FOLLOWER
I20260928 09:04:24.718287 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35649 } }
I20260928 09:04:24.718376 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:24.718416 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:24.718454 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:24.719038 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35649 } }
I20260928 09:04:24.719161 998 leader_election.cc:302] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0956e84ae01d4f0c98c9227ac4b15ff2; no voters:
I20260928 09:04:24.719352 998 leader_election.cc:288] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:24.719517 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:24.719617 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:24.719789 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 0956e84ae01d4f0c98c9227ac4b15ff2, State: Running, Role: LEADER
I20260928 09:04:24.719928 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35649 } }
I20260928 09:04:24.720335 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35649 } } }
I20260928 09:04:24.720451 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:24.720577 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0956e84ae01d4f0c98c9227ac4b15ff2. Latest consensus state: current_term: 1 leader_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0956e84ae01d4f0c98c9227ac4b15ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35649 } } }
I20260928 09:04:24.720638 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:24.720742 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:24.721288 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:04:24 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:24.723134 1012 catalog_manager.cc:1347] Generated new cluster ID: cf3f109e4ec0499da910895f876097b9
I20260928 09:04:24.723189 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:24.725642 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:24.714830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.0.98.62:35649)
0928 09:04:24.715025 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:24.715036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:24.715071 (+ 35us) server_negotiation.cc:407] Connection header received
0928 09:04:24.715654 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:24.715663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:24.715728 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:24.715900 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:24.716583 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:24.717501 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:24.718411 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:24.718905 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:24.719031 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:24.722127 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:24.722131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:24.722132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:24.722146 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:24.724022 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:24.724408 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:24.724411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:24.724412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:24.724511 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:24.724721 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:24.724723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:24.724723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:24.724975 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:24.725155 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:24.725236 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 09:04:24.725380 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260928 09:04:24.741588 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:24.742321 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:24.749456 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0956e84ae01d4f0c98c9227ac4b15ff2: Generated new TSK 0
I20260928 09:04:24.749673 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:24.781175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35649
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:24.865783 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:24.866068 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:24.866111 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:24.867331 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:24.867400 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:24.867444 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:24.868908 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:24.869196 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:24.869428 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:24.874449 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:24.874543 1024 server_base.cc:1034] running on GCE node
W20260928 09:04:24.874449 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:24.874490 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:24.875007 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:24.875249 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:24.876397 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790586264876383 us; error 30 us; skew 500 ppm
Sep 28 09:04:24 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:24.878988 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:24.879500 1024 webserver.cc:460] Webserver started at http://127.0.98.1:37927/ using document root <none> and password file <none>
I20260928 09:04:24.879757 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:24.879824 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:24.880007 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:24.881105 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0"
format_stamp: "Formatted at 2026-09-28 09:04:24 on dist-test-slave-smns"
server_key: "0b4feaf01353513d1861f140767789a5"
server_key_iv: "19c80c3ab2d93f17b03a0941bb2df50e"
server_key_version: "encryptionkey@0"
I20260928 09:04:24.881491 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0"
format_stamp: "Formatted at 2026-09-28 09:04:24 on dist-test-slave-smns"
server_key: "0b4feaf01353513d1861f140767789a5"
server_key_iv: "19c80c3ab2d93f17b03a0941bb2df50e"
server_key_version: "encryptionkey@0"
I20260928 09:04:24.882860 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:24.883646 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:24.883870 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:24.883965 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0"
format_stamp: "Formatted at 2026-09-28 09:04:24 on dist-test-slave-smns"
server_key: "0b4feaf01353513d1861f140767789a5"
server_key_iv: "19c80c3ab2d93f17b03a0941bb2df50e"
server_key_version: "encryptionkey@0"
I20260928 09:04:24.884037 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:24.920992 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:24.923094 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:24.923266 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:24.923530 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:24.923888 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:24.923942 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:24.923978 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:24.924005 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:24.930009 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43735
I20260928 09:04:24.930055 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43735 every 8 connection(s)
I20260928 09:04:24.930377 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:04:24 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:24.936540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1024
I20260928 09:04:24.936661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:24.936836 392 external_mini_cluster.cc:1428] Setting key 2165c0da39797b17324bdb6a5c5da38f
I20260928 09:04:24.940014 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:24.931086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35649 (local address 127.0.98.1:46325)
0928 09:04:24.931268 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:24.931304 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:24.931819 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:24.932078 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:24.932082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:24.932697 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:24.932897 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:24.932903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:24.933696 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:24.933697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:24.934511 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:24.934513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:24.934944 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:24.934946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:24.935079 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:24.935559 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:24.935568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:24.936961 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:24.938631 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:24.938637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:24.938639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:24.938951 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:24.939170 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:24.939171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:24.939171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:24.939270 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:24.939543 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:24.939548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:24.939660 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 09:04:24.939740 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:04:24.939853 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260928 09:04:24.940027 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:24.931117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46325 (local address 127.0.98.62:35649)
0928 09:04:24.931166 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:24.931168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:24.931397 (+ 229us) server_negotiation.cc:407] Connection header received
0928 09:04:24.931931 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:24.931932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:24.931944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:24.931998 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:24.933024 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:24.933542 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:24.934609 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:24.934816 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:24.934936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:24.937062 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:24.937064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:24.937064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:24.937071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:24.938551 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:24.939026 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:24.939028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:24.939028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:24.939099 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:24.939333 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:24.939334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:24.939335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:24.939478 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:24.939546 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:24.939785 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 09:04:24.939885 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:04:24.940498 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35649
I20260928 09:04:24.940598 1153 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:24.940781 1153 heartbeater.cc:511] Master 127.0.98.62:35649 requested a full tablet report, sending...
I20260928 09:04:24.941310 962 ts_manager.cc:194] Registered new tserver with Master: 1aa12b1755ff4ecfb41208c6f8ec66d0 (127.0.98.1:43735)
I20260928 09:04:24.942011 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46325
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:24.965685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35649
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:25.047590 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:25.047859 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:25.047899 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:25.049158 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:25.049248 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:25.049297 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:25.050931 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:25.051221 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:25.051465 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:25.056459 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:25.056481 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:25.056510 1158 server_base.cc:1034] running on GCE node
W20260928 09:04:25.056466 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:25.056872 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:25.057135 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:25.058292 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790586265058283 us; error 44 us; skew 500 ppm
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:25.060705 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:25.061252 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43067/ using document root <none> and password file <none>
I20260928 09:04:25.061492 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:25.061578 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:25.061734 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:25.062786 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "a562fde1511548499122803347ddda21"
format_stamp: "Formatted at 2026-09-28 09:04:25 on dist-test-slave-smns"
server_key: "5790de3402053ee24f128f7c5f1c6dca"
server_key_iv: "4081d85dcb471db999e4c796ad86fba8"
server_key_version: "encryptionkey@0"
I20260928 09:04:25.063155 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a562fde1511548499122803347ddda21"
format_stamp: "Formatted at 2026-09-28 09:04:25 on dist-test-slave-smns"
server_key: "5790de3402053ee24f128f7c5f1c6dca"
server_key_iv: "4081d85dcb471db999e4c796ad86fba8"
server_key_version: "encryptionkey@0"
I20260928 09:04:25.064497 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:04:25.065299 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:25.065492 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:25.065601 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "a562fde1511548499122803347ddda21"
format_stamp: "Formatted at 2026-09-28 09:04:25 on dist-test-slave-smns"
server_key: "5790de3402053ee24f128f7c5f1c6dca"
server_key_iv: "4081d85dcb471db999e4c796ad86fba8"
server_key_version: "encryptionkey@0"
I20260928 09:04:25.065691 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:25.087649 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:25.089708 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:25.089879 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:25.090126 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:25.090452 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:25.090507 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:25.090571 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:25.090601 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:25.096349 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34823
I20260928 09:04:25.096397 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34823 every 8 connection(s)
I20260928 09:04:25.096715 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:25.100627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1158
I20260928 09:04:25.100735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:25.100888 392 external_mini_cluster.cc:1428] Setting key 7dbaf41e282f14c86538a556753647e0
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:25.106532 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.097430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41639 (local address 127.0.98.62:35649)
0928 09:04:25.097478 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.097480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.097726 (+ 246us) server_negotiation.cc:407] Connection header received
0928 09:04:25.098176 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.098178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.098189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.098242 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:25.099203 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.099752 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.100789 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.101046 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.101114 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.103495 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:25.103496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:25.103497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:25.103505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:25.105062 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.105516 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.105518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.105518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.105601 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.105868 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.105869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.105869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.105992 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:25.106069 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.106358 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.106436 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:04:25.106787 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.097397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35649 (local address 127.0.98.2:41639)
0928 09:04:25.097566 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.097633 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.098094 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.098300 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.098303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.098934 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:25.099126 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.099130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.099870 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.099871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.100687 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.100689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.101171 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.101172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.101314 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.101863 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:25.101872 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:25.103397 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:25.105167 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:25.105174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:25.105176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:25.105420 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:25.105674 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:25.105675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:25.105675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:25.105796 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:25.106060 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:25.106066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:25.106212 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.106334 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.106494 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260928 09:04:25.107290 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35649
I20260928 09:04:25.107388 1287 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:25.107594 1287 heartbeater.cc:511] Master 127.0.98.62:35649 requested a full tablet report, sending...
I20260928 09:04:25.107985 962 ts_manager.cc:194] Registered new tserver with Master: a562fde1511548499122803347ddda21 (127.0.98.2:34823)
I20260928 09:04:25.108395 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41639
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:25.129328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35649
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:25.211779 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:25.211992 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:25.212016 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:25.213114 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:25.213150 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:25.213174 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:25.214591 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:25.214766 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:25.214944 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:25.219945 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:25.219970 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:25.220022 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:25.220037 1292 server_base.cc:1034] running on GCE node
I20260928 09:04:25.220360 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:25.220597 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:25.221735 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790586265221698 us; error 51 us; skew 500 ppm
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:25.224190 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:25.224661 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42757/ using document root <none> and password file <none>
I20260928 09:04:25.224841 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:25.224881 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:25.224972 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:25.226141 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "7a36fe1952384c99a365e802ecb754be"
format_stamp: "Formatted at 2026-09-28 09:04:25 on dist-test-slave-smns"
server_key: "06c370d421b70e18573bd242755dc502"
server_key_iv: "e57f6d5a0ed43bf074a253b6cf02e4ce"
server_key_version: "encryptionkey@0"
I20260928 09:04:25.226449 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7a36fe1952384c99a365e802ecb754be"
format_stamp: "Formatted at 2026-09-28 09:04:25 on dist-test-slave-smns"
server_key: "06c370d421b70e18573bd242755dc502"
server_key_iv: "e57f6d5a0ed43bf074a253b6cf02e4ce"
server_key_version: "encryptionkey@0"
I20260928 09:04:25.227715 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:25.228433 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:25.228657 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:25.228750 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "7a36fe1952384c99a365e802ecb754be"
format_stamp: "Formatted at 2026-09-28 09:04:25 on dist-test-slave-smns"
server_key: "06c370d421b70e18573bd242755dc502"
server_key_iv: "e57f6d5a0ed43bf074a253b6cf02e4ce"
server_key_version: "encryptionkey@0"
I20260928 09:04:25.228806 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:25.237949 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:25.240244 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:25.240360 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:25.240540 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:25.240798 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:25.240828 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:25.240850 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:25.240863 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:25.246734 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46143
I20260928 09:04:25.246878 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46143 every 8 connection(s)
I20260928 09:04:25.247126 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:25.253976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1292
I20260928 09:04:25.254081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:25.254280 392 external_mini_cluster.cc:1428] Setting key 2ce95afe0b9d24327d11f8685f77ef28
I20260928 09:04:25.256947 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.247957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59373 (local address 127.0.98.62:35649)
0928 09:04:25.248024 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.248026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.248155 (+ 129us) server_negotiation.cc:407] Connection header received
0928 09:04:25.248678 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.248679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.248691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.248742 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:25.249906 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.250423 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.251443 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.251658 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.251745 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.253903 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:25.253904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:25.253905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:25.253913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:25.255547 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.255947 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.255948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.255949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.256009 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.256281 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.256282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.256282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.256400 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:25.256480 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.256756 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.256856 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:04:25.257138 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.247827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35649 (local address 127.0.98.3:59373)
0928 09:04:25.248027 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.248074 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.248566 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.248819 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.248823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.249411 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:25.249708 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.249714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.250532 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.250533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.251338 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.251340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.251760 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.251761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.251883 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.252409 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:25.252417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:25.253806 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:25.255637 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:25.255643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:25.255645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:25.255870 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:25.256080 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:25.256081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:25.256081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:25.256191 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:25.256468 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:25.256474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:25.256623 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.256735 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.256901 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
I20260928 09:04:25.257676 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35649
I20260928 09:04:25.257762 1421 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:25.257968 1421 heartbeater.cc:511] Master 127.0.98.62:35649 requested a full tablet report, sending...
I20260928 09:04:25.258356 962 ts_manager.cc:194] Registered new tserver with Master: 7a36fe1952384c99a365e802ecb754be (127.0.98.3:46143)
I20260928 09:04:25.258690 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59373
I20260928 09:04:25.266264 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:04:25.274134 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.267959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.0.98.62:35649)
0928 09:04:25.268019 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.268020 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.268164 (+ 144us) server_negotiation.cc:407] Connection header received
0928 09:04:25.268190 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.268190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.268200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.268266 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:25.268866 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.269468 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.270250 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.270448 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.270506 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.271622 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:25.271623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:25.271624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:25.271629 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:25.273051 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.273355 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.273356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.273356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.273405 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.273665 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.273667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.273667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.273816 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:25.273931 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.274009 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.274071 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:04:25.276163 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:25.277740 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:25.284248 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.280113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43735 (local address 127.0.0.1:58880)
0928 09:04:25.280411 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.280422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.280471 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.280784 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.280785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.280787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:25.280838 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.280839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.282022 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.282023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.283266 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.283267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.283927 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.283927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.284004 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.284017 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.284099 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.284149 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":11,"thread_start_us":81,"threads_started":1}
I20260928 09:04:25.284461 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.280185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.0.98.2:34823)
0928 09:04:25.280336 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.280340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.280352 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:25.280452 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.280455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.280471 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.280550 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:25.280738 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.281746 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.283292 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.284026 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.284095 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.284146 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.284244 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.284280 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260928 09:04:25.284642 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.280446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.0.98.1:43735)
0928 09:04:25.280619 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.280623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.280632 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:25.280667 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.280669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.280682 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.280940 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:25.281015 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.281927 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.283344 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.283816 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.283925 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.283994 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.284505 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.284564 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260928 09:04:25.284770 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.280102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34823 (local address 127.0.0.1:39500)
0928 09:04:25.280245 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.280286 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.280350 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.280570 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.280573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.280575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:25.280638 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.280639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.281870 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.281872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.283202 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.283204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.284110 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.284110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.284171 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.284174 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.284230 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.284267 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":8,"thread_start_us":52,"threads_started":1}
I20260928 09:04:25.285120 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.280303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46143 (local address 127.0.0.1:47720)
0928 09:04:25.280699 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.280709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.280754 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.281816 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.281817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.281820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:25.281864 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.281865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.282713 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.282714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.283827 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.283829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.284878 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.284890 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.284958 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.284961 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.285014 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.285053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":71,"threads_started":1}
I20260928 09:04:25.285133 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.280695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.98.3:46143)
0928 09:04:25.281661 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.281666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.281676 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:25.281712 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.281714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.281728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.281804 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:25.281936 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.282616 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.283937 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.284720 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.284789 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.284842 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.285027 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.285065 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":71,"threads_started":1}
I20260928 09:04:25.285357 1221 tablet_service.cc:1467] Processing CreateTablet for tablet bdedba13e5a244cb81c161b7d2a0a503 (DEFAULT_TABLE table=test-table [id=36343187fab949b084ddf8ddd347f58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:25.285315 1087 tablet_service.cc:1467] Processing CreateTablet for tablet bdedba13e5a244cb81c161b7d2a0a503 (DEFAULT_TABLE table=test-table [id=36343187fab949b084ddf8ddd347f58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:25.285708 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdedba13e5a244cb81c161b7d2a0a503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:25.285708 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdedba13e5a244cb81c161b7d2a0a503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:25.289026 1438 tablet_bootstrap.cc:492] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: Bootstrap starting.
I20260928 09:04:25.289420 1355 tablet_service.cc:1467] Processing CreateTablet for tablet bdedba13e5a244cb81c161b7d2a0a503 (DEFAULT_TABLE table=test-table [id=36343187fab949b084ddf8ddd347f58f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:25.289685 1438 tablet_bootstrap.cc:654] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:25.289754 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdedba13e5a244cb81c161b7d2a0a503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:25.290184 1438 log.cc:824] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:25.290530 1439 tablet_bootstrap.cc:492] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: Bootstrap starting.
I20260928 09:04:25.290894 1438 tablet_bootstrap.cc:492] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: No bootstrap required, opened a new log
I20260928 09:04:25.290946 1438 ts_tablet_manager.cc:1397] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 09:04:25.292269 1441 tablet_bootstrap.cc:492] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: Bootstrap starting.
I20260928 09:04:25.292949 1441 tablet_bootstrap.cc:654] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:25.292959 1438 raft_consensus.cc:348] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.293159 1438 raft_consensus.cc:374] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:25.293205 1438 raft_consensus.cc:729] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a562fde1511548499122803347ddda21, State: Initialized, Role: FOLLOWER
I20260928 09:04:25.293322 1438 consensus_queue.cc:260] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.293478 1441 log.cc:824] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:25.293620 1438 ts_tablet_manager.cc:1428] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:04:25.293617 1287 heartbeater.cc:503] Master 127.0.98.62:35649 was elected leader, sending a full tablet report...
I20260928 09:04:25.294229 1439 tablet_bootstrap.cc:654] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:25.294332 1441 tablet_bootstrap.cc:492] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: No bootstrap required, opened a new log
I20260928 09:04:25.294404 1441 ts_tablet_manager.cc:1397] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:25.294713 1439 log.cc:824] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:25.295414 1439 tablet_bootstrap.cc:492] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: No bootstrap required, opened a new log
I20260928 09:04:25.295495 1439 ts_tablet_manager.cc:1397] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 09:04:25.296175 1441 raft_consensus.cc:348] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.296335 1441 raft_consensus.cc:374] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:25.296360 1441 raft_consensus.cc:729] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36fe1952384c99a365e802ecb754be, State: Initialized, Role: FOLLOWER
I20260928 09:04:25.296496 1441 consensus_queue.cc:260] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.296782 1441 ts_tablet_manager.cc:1428] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:25.296801 1421 heartbeater.cc:503] Master 127.0.98.62:35649 was elected leader, sending a full tablet report...
I20260928 09:04:25.297204 1439 raft_consensus.cc:348] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.297344 1439 raft_consensus.cc:374] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:25.297389 1439 raft_consensus.cc:729] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa12b1755ff4ecfb41208c6f8ec66d0, State: Initialized, Role: FOLLOWER
I20260928 09:04:25.297497 1439 consensus_queue.cc:260] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.297760 1439 ts_tablet_manager.cc:1428] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:25.297806 1153 heartbeater.cc:503] Master 127.0.98.62:35649 was elected leader, sending a full tablet report...
W20260928 09:04:25.347288 1288 tablet.cc:2369] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:04:25.431138 1154 tablet.cc:2369] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:25.440088 1446 raft_consensus.cc:482] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:25.440217 1446 raft_consensus.cc:504] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.440635 1446 leader_election.cc:288] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a562fde1511548499122803347ddda21 (127.0.98.2:34823), 7a36fe1952384c99a365e802ecb754be (127.0.98.3:46143)
I20260928 09:04:25.443614 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.440831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46143 (local address 127.0.98.1:45327)
0928 09:04:25.440936 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.440957 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.441035 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.441192 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.441193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.441195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:25.441269 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.441271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.442004 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.442005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.442782 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.442784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.443332 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.443333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.443405 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.443409 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.443475 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.443522 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260928 09:04:25.443601 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.440974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45327 (local address 127.0.98.3:46143)
0928 09:04:25.441024 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.441026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.441035 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:25.441092 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.441093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.441104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.441175 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:25.441341 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.441884 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.442880 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.443246 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.443331 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.443341 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.443493 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.443539 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 09:04:25.443738 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.440912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56669 (local address 127.0.98.2:34823)
0928 09:04:25.440976 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.440979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.441026 (+ 47us) server_negotiation.cc:407] Connection header received
0928 09:04:25.441148 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.441149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.441162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.441230 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:25.441376 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.441882 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.442973 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.443376 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.443462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.443472 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.443638 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.443676 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:04:25.443756 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.440831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34823 (local address 127.0.98.1:56669)
0928 09:04:25.440954 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:25.440965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:25.441068 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:25.441234 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:25.441235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:25.441237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:25.441307 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.441309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.441994 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.441995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.442844 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:25.442847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.443462 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.443463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:25.443547 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.443550 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:25.443628 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 09:04:25.443666 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":68,"thread_start_us":72,"threads_started":1}
I20260928 09:04:25.444096 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdedba13e5a244cb81c161b7d2a0a503" candidate_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36fe1952384c99a365e802ecb754be" is_pre_election: true
I20260928 09:04:25.444147 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdedba13e5a244cb81c161b7d2a0a503" candidate_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a562fde1511548499122803347ddda21" is_pre_election: true
I20260928 09:04:25.444218 1375 raft_consensus.cc:2454] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa12b1755ff4ecfb41208c6f8ec66d0 in term 0.
I20260928 09:04:25.444317 1241 raft_consensus.cc:2454] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1aa12b1755ff4ecfb41208c6f8ec66d0 in term 0.
I20260928 09:04:25.444481 1043 leader_election.cc:302] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa12b1755ff4ecfb41208c6f8ec66d0, 7a36fe1952384c99a365e802ecb754be; no voters:
I20260928 09:04:25.444653 1446 raft_consensus.cc:2781] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:25.444697 1446 raft_consensus.cc:482] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:25.444713 1446 raft_consensus.cc:3037] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:25.445399 1446 raft_consensus.cc:504] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.445562 1446 leader_election.cc:288] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [CANDIDATE]: Term 1 election: Requested vote from peers a562fde1511548499122803347ddda21 (127.0.98.2:34823), 7a36fe1952384c99a365e802ecb754be (127.0.98.3:46143)
I20260928 09:04:25.445735 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdedba13e5a244cb81c161b7d2a0a503" candidate_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36fe1952384c99a365e802ecb754be"
I20260928 09:04:25.445739 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdedba13e5a244cb81c161b7d2a0a503" candidate_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a562fde1511548499122803347ddda21"
I20260928 09:04:25.445818 1241 raft_consensus.cc:3037] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:25.445818 1375 raft_consensus.cc:3037] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:25.446748 1241 raft_consensus.cc:2454] T bdedba13e5a244cb81c161b7d2a0a503 P a562fde1511548499122803347ddda21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa12b1755ff4ecfb41208c6f8ec66d0 in term 1.
I20260928 09:04:25.446748 1375 raft_consensus.cc:2454] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1aa12b1755ff4ecfb41208c6f8ec66d0 in term 1.
I20260928 09:04:25.446983 1042 leader_election.cc:302] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa12b1755ff4ecfb41208c6f8ec66d0, a562fde1511548499122803347ddda21; no voters:
I20260928 09:04:25.447077 1446 raft_consensus.cc:2781] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:25.447224 1446 raft_consensus.cc:686] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [term 1 LEADER]: Becoming Leader. State: Replica: 1aa12b1755ff4ecfb41208c6f8ec66d0, State: Running, Role: LEADER
I20260928 09:04:25.447317 1446 consensus_queue.cc:237] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } }
I20260928 09:04:25.448176 962 catalog_manager.cc:5495] T bdedba13e5a244cb81c161b7d2a0a503 P 1aa12b1755ff4ecfb41208c6f8ec66d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1aa12b1755ff4ecfb41208c6f8ec66d0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aa12b1755ff4ecfb41208c6f8ec66d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a562fde1511548499122803347ddda21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36fe1952384c99a365e802ecb754be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46143 } health_report { overall_health: UNKNOWN } } }
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:25.474169 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.466094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.0.98.1:43735)
0928 09:04:25.466175 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.466177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.466317 (+ 140us) server_negotiation.cc:407] Connection header received
0928 09:04:25.466419 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.466420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.466438 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.466510 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:25.467193 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.467747 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.468556 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.468840 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.468922 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.471394 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:25.471412 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:25.471413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:25.471420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:25.472881 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.473284 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.473289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.473289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.473363 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.473601 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.473602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.473602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.473760 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:25.473918 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.473982 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.474061 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:25.496685 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.489337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.0.98.1:43735)
0928 09:04:25.489412 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.489414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.489493 (+ 79us) server_negotiation.cc:407] Connection header received
0928 09:04:25.489617 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.489618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.489628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.489700 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:25.490417 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.490945 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.491717 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.491922 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.492009 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.494284 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:25.494286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:25.494286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:25.494292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:25.495694 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.496056 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.496057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.496057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.496108 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.496274 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.496274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.496274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.496361 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:25.496432 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.496562 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.496619 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 09:04:25.497695 1422 tablet.cc:2369] T bdedba13e5a244cb81c161b7d2a0a503 P 7a36fe1952384c99a365e802ecb754be: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:25.518112 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:25.511722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.0.98.1:43735)
0928 09:04:25.511789 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:25.511791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:25.511864 (+ 73us) server_negotiation.cc:407] Connection header received
0928 09:04:25.511913 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:25.511913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:25.511921 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:25.511980 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:25.512482 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.512932 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.513747 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:25.513923 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:25.513981 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:25.516010 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:25.516012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:25.516012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:25.516018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:25.517132 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.517436 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.517437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.517437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.517488 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:25.517701 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:25.517702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:25.517702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:25.517794 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:25.517868 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:25.517958 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 09:04:25.518014 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 09:04:25.518622 1067 tablet_service.cc:2908] Rejecting scan request for tablet bdedba13e5a244cb81c161b7d2a0a503: Uninitialized: clean time has not yet been initialized
I20260928 09:04:25.519596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1024
I20260928 09:04:25.524642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1158
I20260928 09:04:25.528965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1292
I20260928 09:04:25.533051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 930
2026-09-28T09:04:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2092 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[1463](info): set up 2 sockets
Sep 28 09:04:25 dist-test-slave-smns krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260928 09:04:26.569114 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:26 dist-test-slave-smns krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:26.688951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42925
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:26.769668 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:26.769959 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:26.769999 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:26.771166 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:26.771209 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 09:04:26.771221 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:26.771235 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:26.771247 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:26.771301 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:26.772830 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42925
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:26.773079 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:26.773314 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:26.778147 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:26.778178 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:26.778270 1476 server_base.cc:1034] running on GCE node
W20260928 09:04:26.778160 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:26.778661 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:26.778911 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:26.780049 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790586266780029 us; error 41 us; skew 500 ppm
Sep 28 09:04:26 dist-test-slave-smns krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:26.782646 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:26.783133 1476 webserver.cc:460] Webserver started at http://127.0.98.62:35715/ using document root <none> and password file <none>
I20260928 09:04:26.783378 1476 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:26.783452 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:26.783608 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:26.784687 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "46b5f1ba834241c58169e20388e504ed"
format_stamp: "Formatted at 2026-09-28 09:04:26 on dist-test-slave-smns"
server_key: "59231207fde9344a76fea001e2eb3a40"
server_key_iv: "049230a44e353762d5670beec1bdb3b1"
server_key_version: "encryptionkey@0"
I20260928 09:04:26.785043 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "46b5f1ba834241c58169e20388e504ed"
format_stamp: "Formatted at 2026-09-28 09:04:26 on dist-test-slave-smns"
server_key: "59231207fde9344a76fea001e2eb3a40"
server_key_iv: "049230a44e353762d5670beec1bdb3b1"
server_key_version: "encryptionkey@0"
I20260928 09:04:26.786406 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:26.787132 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:26.787303 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:26.787402 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "46b5f1ba834241c58169e20388e504ed"
format_stamp: "Formatted at 2026-09-28 09:04:26 on dist-test-slave-smns"
server_key: "59231207fde9344a76fea001e2eb3a40"
server_key_iv: "049230a44e353762d5670beec1bdb3b1"
server_key_version: "encryptionkey@0"
I20260928 09:04:26.787478 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:26.796353 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:26.798141 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:26.798305 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:26.802887 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42925
I20260928 09:04:26.802933 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42925 every 8 connection(s)
I20260928 09:04:26.803248 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:26.803515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1476
I20260928 09:04:26.803618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:26.803788 392 external_mini_cluster.cc:1428] Setting key 7309382dd7c31e605cd48a2bc8c1106a
I20260928 09:04:26.803977 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:26.807353 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: Bootstrap starting.
I20260928 09:04:26.808055 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:26.808339 1554 log.cc:824] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:26.809130 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: No bootstrap required, opened a new log
Sep 28 09:04:26 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:26.811105 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b5f1ba834241c58169e20388e504ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260928 09:04:26.811290 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:26.811331 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b5f1ba834241c58169e20388e504ed, State: Initialized, Role: FOLLOWER
I20260928 09:04:26.811470 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b5f1ba834241c58169e20388e504ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260928 09:04:26.811575 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:26.811645 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:26.811698 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:26.812424 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b5f1ba834241c58169e20388e504ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260928 09:04:26.812541 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b5f1ba834241c58169e20388e504ed; no voters:
I20260928 09:04:26.812700 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:26.812773 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:26.812942 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:26.812986 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [term 1 LEADER]: Becoming Leader. State: Replica: 46b5f1ba834241c58169e20388e504ed, State: Running, Role: LEADER
I20260928 09:04:26.813196 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b5f1ba834241c58169e20388e504ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260928 09:04:26.813563 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b5f1ba834241c58169e20388e504ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b5f1ba834241c58169e20388e504ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } } }
I20260928 09:04:26.813638 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:26.813905 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b5f1ba834241c58169e20388e504ed. Latest consensus state: current_term: 1 leader_uuid: "46b5f1ba834241c58169e20388e504ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b5f1ba834241c58169e20388e504ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } } }
I20260928 09:04:26.814033 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:26.814201 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:26.814723 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:26.815454 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:26.804300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.0.98.62:42925)
0928 09:04:26.804473 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:26.804482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:26.804518 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:04:26.805092 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:26.805099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:26.805149 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:26.805263 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:26.806105 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:26.807046 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:26.808074 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:26.808567 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:26.808649 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:26.811102 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:26.811105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:26.811106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:26.811117 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:26.813276 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:26.813869 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:26.813873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:26.813873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:26.813980 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:26.814196 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:26.814197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:26.814197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:26.814537 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:26.814677 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:26.815100 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 09:04:26.815231 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260928 09:04:26.816329 1568 catalog_manager.cc:1347] Generated new cluster ID: 5d8a26a2d36a4e1bb2a1a5789b7652fb
I20260928 09:04:26.816377 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:26.850634 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:26.851362 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:26.856097 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: Generated new TSK 0
I20260928 09:04:26.856280 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:26.887867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42925
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:26.969990 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:26.970189 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:26.970214 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:26.971419 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:26.971460 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:26.971481 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:26.971499 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:26.973349 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:26.973639 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:26.973905 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:26.979489 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:26.979554 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:26.979562 1580 server_base.cc:1034] running on GCE node
W20260928 09:04:26.979489 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:26.979954 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:26.980257 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:26.981451 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790586266981414 us; error 46 us; skew 500 ppm
Sep 28 09:04:26 dist-test-slave-smns krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:26.984191 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:26.984586 1580 webserver.cc:460] Webserver started at http://127.0.98.1:43861/ using document root <none> and password file <none>
I20260928 09:04:26.984797 1580 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:26.984845 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:26.984985 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:26.986104 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "4cad8c4d55a04eddbdd62b0835d79e0c"
format_stamp: "Formatted at 2026-09-28 09:04:26 on dist-test-slave-smns"
server_key: "48a27284d6b415bc1f14a30ad311b054"
server_key_iv: "564d6bdd7a755cb6f774f917a6ce6985"
server_key_version: "encryptionkey@0"
I20260928 09:04:26.986469 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4cad8c4d55a04eddbdd62b0835d79e0c"
format_stamp: "Formatted at 2026-09-28 09:04:26 on dist-test-slave-smns"
server_key: "48a27284d6b415bc1f14a30ad311b054"
server_key_iv: "564d6bdd7a755cb6f774f917a6ce6985"
server_key_version: "encryptionkey@0"
I20260928 09:04:26.987843 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:26.988567 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:26.988754 1580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:26.988834 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "4cad8c4d55a04eddbdd62b0835d79e0c"
format_stamp: "Formatted at 2026-09-28 09:04:26 on dist-test-slave-smns"
server_key: "48a27284d6b415bc1f14a30ad311b054"
server_key_iv: "564d6bdd7a755cb6f774f917a6ce6985"
server_key_version: "encryptionkey@0"
I20260928 09:04:26.988880 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:27.012400 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:27.015182 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:27.015340 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:27.015880 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:27.015952 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.016021 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:27.016052 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.023026 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46267
I20260928 09:04:27.023092 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46267 every 8 connection(s)
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:27.023406 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:27.026584 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.016505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52819 (local address 127.0.98.62:42925)
0928 09:04:27.016553 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.016555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.016812 (+ 257us) server_negotiation.cc:407] Connection header received
0928 09:04:27.017380 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.017382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.017395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.017449 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.018324 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.019154 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.020766 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.021137 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.021237 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.023988 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.023989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.023990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.023996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.025306 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.025846 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.025846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.025847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.025895 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.026096 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.026096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.026096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.026191 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.026237 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.026451 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.026519 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:27.026840 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.016527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.98.1:52819)
0928 09:04:27.016689 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.016725 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.017270 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.017529 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.017534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.017914 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.018233 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.018240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.019247 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.019248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.020674 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.020676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.021242 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.021243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.021382 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.021953 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.021964 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.023888 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.025383 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.025389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.025390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.025691 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.025955 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.025955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.025956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.026055 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.026240 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.026244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.026360 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.026433 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.026550 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 09:04:27.027412 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42925
I20260928 09:04:27.027490 1711 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:27.027678 1711 heartbeater.cc:511] Master 127.0.98.62:42925 requested a full tablet report, sending...
I20260928 09:04:27.028184 1507 ts_manager.cc:194] Registered new tserver with Master: 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267)
I20260928 09:04:27.028926 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52819
I20260928 09:04:27.033499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1580
I20260928 09:04:27.033619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:27.033823 392 external_mini_cluster.cc:1428] Setting key 628858aefc9e3f96353e8920f93b9a7e
I20260928 09:04:27.034107 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.028513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35315 (local address 127.0.98.62:42925)
0928 09:04:27.028548 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.028550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.028679 (+ 129us) server_negotiation.cc:407] Connection header received
0928 09:04:27.028751 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.028752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.028761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.028807 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.029162 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.029758 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.030365 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.030541 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.030584 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.031507 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.031508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.031509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.031515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.032968 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.033259 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.033260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.033260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.033341 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.033520 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.033521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.033521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.033692 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.033774 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.033936 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.034019 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:04:27.034168 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.028450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.98.1:35315)
0928 09:04:27.028613 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.028622 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.028694 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.028819 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.028819 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.029022 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.029085 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.029087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.029823 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.029824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.030298 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.030299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.030629 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.030630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.030694 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.031129 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.031134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.031405 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.033045 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.033045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.033046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.033196 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.033409 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.033409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.033409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.033458 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.033763 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.033764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.033801 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.033855 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.034027 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:27.060142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42925
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:27.139567 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:27.139799 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:27.139837 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:27.140983 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:27.141050 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:27.141085 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:27.141124 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:27.142611 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:27.142877 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:27.143097 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:04:27.148087 1720 server_base.cc:1034] running on GCE node
W20260928 09:04:27.148109 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:27.148046 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:27.148056 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:27.148463 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:27.148660 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:27.149791 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790586267149774 us; error 35 us; skew 500 ppm
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:27.152098 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:27.152561 1720 webserver.cc:460] Webserver started at http://127.0.98.2:44061/ using document root <none> and password file <none>
I20260928 09:04:27.152746 1720 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:27.152786 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:27.152874 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:27.153946 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "723d3c6aa9a948e8a46f34b62b5d23bd"
format_stamp: "Formatted at 2026-09-28 09:04:27 on dist-test-slave-smns"
server_key: "ee81a8d2d3bf19baba505f645417971b"
server_key_iv: "0020564f0eb540979f0d2da165eba62d"
server_key_version: "encryptionkey@0"
I20260928 09:04:27.154304 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "723d3c6aa9a948e8a46f34b62b5d23bd"
format_stamp: "Formatted at 2026-09-28 09:04:27 on dist-test-slave-smns"
server_key: "ee81a8d2d3bf19baba505f645417971b"
server_key_iv: "0020564f0eb540979f0d2da165eba62d"
server_key_version: "encryptionkey@0"
I20260928 09:04:27.155649 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:27.156365 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.156559 1720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:27.156636 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "723d3c6aa9a948e8a46f34b62b5d23bd"
format_stamp: "Formatted at 2026-09-28 09:04:27 on dist-test-slave-smns"
server_key: "ee81a8d2d3bf19baba505f645417971b"
server_key_iv: "0020564f0eb540979f0d2da165eba62d"
server_key_version: "encryptionkey@0"
I20260928 09:04:27.156688 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:27.172430 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:27.174319 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:27.174433 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:27.174983 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:27.175025 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.175047 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:27.175086 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:27.181874 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38283
I20260928 09:04:27.181934 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38283 every 8 connection(s)
I20260928 09:04:27.182260 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:27.184499 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.175381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47767 (local address 127.0.98.62:42925)
0928 09:04:27.175427 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.175430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.175671 (+ 241us) server_negotiation.cc:407] Connection header received
0928 09:04:27.176166 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.176168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.176182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.176267 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.177118 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.177970 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.179004 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.179343 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.179416 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.182041 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.182042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.182043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.182050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.183421 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.183774 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.183775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.183775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.183818 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.184031 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.184031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.184032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.184119 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.184164 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.184364 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.184418 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:04:27.184631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1720
I20260928 09:04:27.184720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:27.184729 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.175363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.98.2:47767)
0928 09:04:27.175516 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.175564 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.176066 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.176319 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.176323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.176718 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.177011 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.177018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.178069 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.178071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.178933 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.178936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.179460 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.179462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.179606 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.180134 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.180146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.181957 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.183498 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.183503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.183505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.183714 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.183870 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.183871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.183871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.183984 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.184169 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.184172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.184277 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.184353 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.184499 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260928 09:04:27.184911 392 external_mini_cluster.cc:1428] Setting key c4ab82f8f9953390907a754e7e3dbd31
I20260928 09:04:27.185212 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42925
I20260928 09:04:27.185297 1851 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:27.185496 1851 heartbeater.cc:511] Master 127.0.98.62:42925 requested a full tablet report, sending...
I20260928 09:04:27.185859 1508 ts_manager.cc:194] Registered new tserver with Master: 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2:38283)
I20260928 09:04:27.186379 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47767
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:04:27.193533 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.186660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48129 (local address 127.0.98.62:42925)
0928 09:04:27.186693 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.186695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.187470 (+ 775us) server_negotiation.cc:407] Connection header received
0928 09:04:27.187539 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.187539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.187549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.187594 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.187920 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.188368 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.189512 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.189865 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.189919 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.190912 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.190913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.190914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.190919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.192340 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.192784 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.192785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.192785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.192836 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.193035 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.193036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.193036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.193159 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.193210 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.193400 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.193474 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 09:04:27.193620 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.186629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.98.2:48129)
0928 09:04:27.187381 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.187399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.187483 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.187596 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.187596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.187770 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.187818 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.187819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.188469 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.188470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.189433 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.189436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.189946 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.189947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.190023 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.190414 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.190420 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.190833 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.192421 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.192422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.192423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.192642 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.192905 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.192906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.192906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.192973 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.193223 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.193224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.193268 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.193348 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.193467 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":707,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:27.212073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42925
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:27.292716 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:27.292953 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:27.292992 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:27.294148 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:27.294214 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:27.294246 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:27.294286 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:27.295745 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:27.296017 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:27.296257 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:27.301005 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:27.301079 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:27.301038 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:27.301102 1860 server_base.cc:1034] running on GCE node
I20260928 09:04:27.301352 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:27.301600 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:27.302766 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790586267302753 us; error 24 us; skew 500 ppm
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:27.305112 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:27.305476 1860 webserver.cc:460] Webserver started at http://127.0.98.3:40267/ using document root <none> and password file <none>
I20260928 09:04:27.305704 1860 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:27.305750 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:27.305835 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:27.306851 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "04b227592fdc4405ab87ff3fe5abea81"
format_stamp: "Formatted at 2026-09-28 09:04:27 on dist-test-slave-smns"
server_key: "ebc0e64523c41e9b6b57f550209f250a"
server_key_iv: "d67e206191628445cfe835c113370e06"
server_key_version: "encryptionkey@0"
I20260928 09:04:27.307143 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "04b227592fdc4405ab87ff3fe5abea81"
format_stamp: "Formatted at 2026-09-28 09:04:27 on dist-test-slave-smns"
server_key: "ebc0e64523c41e9b6b57f550209f250a"
server_key_iv: "d67e206191628445cfe835c113370e06"
server_key_version: "encryptionkey@0"
I20260928 09:04:27.308429 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:27.309131 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.309319 1860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:27.309405 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "04b227592fdc4405ab87ff3fe5abea81"
format_stamp: "Formatted at 2026-09-28 09:04:27 on dist-test-slave-smns"
server_key: "ebc0e64523c41e9b6b57f550209f250a"
server_key_iv: "d67e206191628445cfe835c113370e06"
server_key_version: "encryptionkey@0"
I20260928 09:04:27.309458 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:27.329169 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:27.331048 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:27.331158 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:27.331614 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:27.331650 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.331676 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:27.331717 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.338286 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37317
I20260928 09:04:27.338375 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37317 every 8 connection(s)
I20260928 09:04:27.338667 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:27.341926 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.332187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57159 (local address 127.0.98.62:42925)
0928 09:04:27.332229 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.332232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.332478 (+ 246us) server_negotiation.cc:407] Connection header received
0928 09:04:27.333043 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.333045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.333059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.333137 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.334068 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.334917 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.336502 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.336830 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.336922 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.339420 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.339421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.339421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.339428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.340655 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.341031 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.341031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.341032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.341083 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.341290 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.341290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.341291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.341399 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.341449 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.341763 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.341833 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:27.342051 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.332184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.98.3:57159)
0928 09:04:27.332355 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.332390 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.332898 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.333231 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.333236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.333675 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.333959 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.333965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.335017 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.335018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.336389 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.336392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.336925 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.336927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.337086 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.337690 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.337706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.339294 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.340737 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.340743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.340745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.340963 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.341143 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.341144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.341144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.341238 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.341459 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.341463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.341627 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.341721 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.341842 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260928 09:04:27.342523 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42925
I20260928 09:04:27.342630 1991 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:27.342805 1991 heartbeater.cc:511] Master 127.0.98.62:42925 requested a full tablet report, sending...
I20260928 09:04:27.343252 1507 ts_manager.cc:194] Registered new tserver with Master: 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.343743 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57159
I20260928 09:04:27.347065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 1860
I20260928 09:04:27.347201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:27.347393 392 external_mini_cluster.cc:1428] Setting key c1eacc6f09ee34b1417ddf7a0ab50f20
I20260928 09:04:27.348376 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:04:27.350479 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.343870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53231 (local address 127.0.98.62:42925)
0928 09:04:27.343935 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.343938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.343945 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:27.343964 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.343965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.343976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.344036 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.344431 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.344965 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.345650 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.345914 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.345979 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.347095 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.347096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.347096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.347101 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.348758 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.349160 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.349160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.349161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.349654 (+ 493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.349928 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.349929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.349929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.350065 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.350144 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.350302 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.350378 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:04:27.350487 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.343529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.98.3:53231)
0928 09:04:27.343638 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.343650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.343702 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.344060 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.344061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.344273 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.344332 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.344334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.345057 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.345058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.345534 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.345535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.346008 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.346009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.346056 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.346521 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.346531 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.346881 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.348839 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.348840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.348841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.349084 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.349719 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.349720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.349720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.349795 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.350136 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.350137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.350184 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.350276 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.350380 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:27.380184 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.371960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.0.98.62:42925)
0928 09:04:27.372022 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.372024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.372034 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:27.372063 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.372064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.372075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.372147 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.372973 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.373885 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.375196 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.375546 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.375636 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.377963 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.377964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.377964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.377970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.379131 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.379459 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.379460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.379461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.379514 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.379745 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.379746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.379746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.379844 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.379919 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.380029 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.380088 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:04:27.381919 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:27.383518 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:27.390686 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.386231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.0.98.2:38283)
0928 09:04:27.386463 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.386467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.386479 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:27.386559 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.386561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.386579 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.386651 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.387044 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.388303 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.389715 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.390280 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.390406 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.390518 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.390551 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.390587 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260928 09:04:27.390679 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.386017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.0.1:48076)
0928 09:04:27.386415 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.386427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.386481 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.386769 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.386773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.386776 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.386880 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.386889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.388397 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.388398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.389622 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.389624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.390395 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.390396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.390464 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.390479 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.390540 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.390586 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":33,"thread_start_us":87,"threads_started":1}
I20260928 09:04:27.390898 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.385987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46267 (local address 127.0.0.1:42920)
0928 09:04:27.386087 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.386128 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.386193 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.387165 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.387166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.387168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.387227 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.387229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.388393 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.388394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.389747 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.389749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.390727 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.390728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.390789 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.390793 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.390821 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.390859 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":51,"thread_start_us":58,"threads_started":1}
I20260928 09:04:27.391363 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.386465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.1:46267)
0928 09:04:27.386642 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.386646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.386659 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:04:27.386747 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.386749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.387056 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.387296 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.387361 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.388301 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.389850 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.390637 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.391129 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.391218 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.391244 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.391281 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260928 09:04:27.391721 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 3b80d774b4a0413fb2fc528855b0b3e4 (DEFAULT_TABLE table=test-table [id=45b44688d17d44f4bc5d40f9b91077a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:27.391877 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.386140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.0.1:47746)
0928 09:04:27.386641 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.386652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.386715 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.387853 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.387854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.387857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.387914 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.387916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.389119 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.389120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.390208 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.390210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.391639 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.391640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.391712 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.391717 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.391776 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.391814 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":94,"threads_started":1}
I20260928 09:04:27.391944 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.386249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.0.98.3:37317)
0928 09:04:27.386414 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.386419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.386705 (+ 286us) server_negotiation.cc:407] Connection header received
0928 09:04:27.386873 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.386876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.386895 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.386984 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.387988 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.389018 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.390289 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.391483 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.391544 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.391608 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.391804 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.391848 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 09:04:27.392105 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b80d774b4a0413fb2fc528855b0b3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.392001 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 3b80d774b4a0413fb2fc528855b0b3e4 (DEFAULT_TABLE table=test-table [id=45b44688d17d44f4bc5d40f9b91077a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:27.392334 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b80d774b4a0413fb2fc528855b0b3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.395279 2014 tablet_bootstrap.cc:492] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Bootstrap starting.
I20260928 09:04:27.395710 2015 tablet_bootstrap.cc:492] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Bootstrap starting.
I20260928 09:04:27.395943 2014 tablet_bootstrap.cc:654] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.396061 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 3b80d774b4a0413fb2fc528855b0b3e4 (DEFAULT_TABLE table=test-table [id=45b44688d17d44f4bc5d40f9b91077a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:27.396278 2014 log.cc:824] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:27.396353 2015 tablet_bootstrap.cc:654] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.396416 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b80d774b4a0413fb2fc528855b0b3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.396649 2015 log.cc:824] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:27.397274 2015 tablet_bootstrap.cc:492] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: No bootstrap required, opened a new log
I20260928 09:04:27.397327 2015 ts_tablet_manager.cc:1397] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:27.398862 2015 raft_consensus.cc:348] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.399000 2015 raft_consensus.cc:374] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.399022 2015 raft_consensus.cc:729] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad8c4d55a04eddbdd62b0835d79e0c, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.399161 2015 consensus_queue.cc:260] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.399322 2018 tablet_bootstrap.cc:492] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Bootstrap starting.
I20260928 09:04:27.399441 2015 ts_tablet_manager.cc:1428] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:04:27.399557 1711 heartbeater.cc:503] Master 127.0.98.62:42925 was elected leader, sending a full tablet report...
I20260928 09:04:27.399822 2014 tablet_bootstrap.cc:492] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: No bootstrap required, opened a new log
I20260928 09:04:27.399915 2014 ts_tablet_manager.cc:1397] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 09:04:27.399943 2018 tablet_bootstrap.cc:654] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.400177 2018 log.cc:824] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:27.400926 2018 tablet_bootstrap.cc:492] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: No bootstrap required, opened a new log
I20260928 09:04:27.400976 2018 ts_tablet_manager.cc:1397] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:27.401801 2014 raft_consensus.cc:348] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.401993 2014 raft_consensus.cc:374] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.402034 2014 raft_consensus.cc:729] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723d3c6aa9a948e8a46f34b62b5d23bd, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.402145 2014 consensus_queue.cc:260] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.402398 2014 ts_tablet_manager.cc:1428] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:27.402494 1851 heartbeater.cc:503] Master 127.0.98.62:42925 was elected leader, sending a full tablet report...
I20260928 09:04:27.402665 2018 raft_consensus.cc:348] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.402814 2018 raft_consensus.cc:374] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.402856 2018 raft_consensus.cc:729] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b227592fdc4405ab87ff3fe5abea81, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.402974 2018 consensus_queue.cc:260] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.403226 2018 ts_tablet_manager.cc:1428] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:27.403244 1991 heartbeater.cc:503] Master 127.0.98.62:42925 was elected leader, sending a full tablet report...
W20260928 09:04:27.432866 1852 tablet.cc:2369] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:04:27.524134 1712 tablet.cc:2369] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:27.552253 2021 raft_consensus.cc:482] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:27.552377 2021 raft_consensus.cc:504] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.552729 2021 leader_election.cc:288] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267), 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.555789 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.553213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38455 (local address 127.0.98.3:37317)
0928 09:04:27.553279 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.553282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.553292 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:27.553371 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.553372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.553389 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.553462 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.553628 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.554109 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.555076 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.555437 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.555517 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.555528 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.555677 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.555711 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:04:27.555819 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.553036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.98.2:38455)
0928 09:04:27.553227 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.553238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.553299 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.553471 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.553472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.553474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.553528 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.553529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.554196 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.554197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.554979 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.554981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.555519 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.555520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.555590 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.555599 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.555670 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.555709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260928 09:04:27.555814 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.553046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39005 (local address 127.0.98.1:46267)
0928 09:04:27.553142 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.553144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.553156 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:27.553188 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.553189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.553212 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.553310 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.553505 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.554032 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.555086 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.555445 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.555540 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.555550 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.555698 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.555737 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 09:04:27.555840 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.552931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46267 (local address 127.0.98.2:39005)
0928 09:04:27.552988 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.553001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.553072 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.553317 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.553318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.553320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.553417 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.553419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.554123 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.554125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.554983 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.554985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.555543 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.555544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.555617 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.555619 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.555694 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.555746 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":5}
I20260928 09:04:27.556219 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b80d774b4a0413fb2fc528855b0b3e4" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b227592fdc4405ab87ff3fe5abea81" is_pre_election: true
I20260928 09:04:27.556227 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b80d774b4a0413fb2fc528855b0b3e4" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" is_pre_election: true
I20260928 09:04:27.556385 1665 raft_consensus.cc:2454] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 0.
I20260928 09:04:27.556391 1945 raft_consensus.cc:2454] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 0.
I20260928 09:04:27.556612 1739 leader_election.cc:302] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cad8c4d55a04eddbdd62b0835d79e0c, 723d3c6aa9a948e8a46f34b62b5d23bd; no voters:
I20260928 09:04:27.556763 2021 raft_consensus.cc:2781] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:27.556819 2021 raft_consensus.cc:482] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:27.556847 2021 raft_consensus.cc:3037] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.557878 2021 raft_consensus.cc:504] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.558028 2021 leader_election.cc:288] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 election: Requested vote from peers 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267), 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.558230 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b80d774b4a0413fb2fc528855b0b3e4" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c"
I20260928 09:04:27.558300 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b80d774b4a0413fb2fc528855b0b3e4" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b227592fdc4405ab87ff3fe5abea81"
I20260928 09:04:27.558317 1665 raft_consensus.cc:3037] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.558403 1945 raft_consensus.cc:3037] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.559008 1665 raft_consensus.cc:2454] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 1.
I20260928 09:04:27.559170 1739 leader_election.cc:302] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cad8c4d55a04eddbdd62b0835d79e0c, 723d3c6aa9a948e8a46f34b62b5d23bd; no voters:
I20260928 09:04:27.559232 1945 raft_consensus.cc:2454] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 1.
I20260928 09:04:27.559278 2021 raft_consensus.cc:2781] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:27.559449 2021 raft_consensus.cc:686] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 LEADER]: Becoming Leader. State: Replica: 723d3c6aa9a948e8a46f34b62b5d23bd, State: Running, Role: LEADER
I20260928 09:04:27.559581 2021 consensus_queue.cc:237] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } }
I20260928 09:04:27.560418 1507 catalog_manager.cc:5495] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: HEALTHY } } }
I20260928 09:04:27.571669 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.569110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.98.2:38283)
0928 09:04:27.569167 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.569169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.569178 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:27.569276 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.569277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.569302 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.569375 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:27.569522 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.570013 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.570809 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.571011 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.571096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.571304 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.571413 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.571504 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.571554 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.571576 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:04:27.573647 1945 raft_consensus.cc:1261] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Refusing update from remote peer 723d3c6aa9a948e8a46f34b62b5d23bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:27.573647 1665 raft_consensus.cc:1261] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Refusing update from remote peer 723d3c6aa9a948e8a46f34b62b5d23bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:27.573987 2021 consensus_queue.cc:1035] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:27.574151 2024 consensus_queue.cc:1035] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:27.576385 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334241351978344448 to 7334241351924473856. Current Snapshot: MvccSnapshot[applied={T|T < 7334241351978344448}]
I20260928 09:04:27.576743 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334241351978344448 to 7334241351924473856. Current Snapshot: MvccSnapshot[applied={T|T < 7334241351978344448}]
I20260928 09:04:27.576989 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334241351978344448 to 7334241351924473856. Current Snapshot: MvccSnapshot[applied={T|T < 7334241351978344448}]
I20260928 09:04:27.580397 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:27.580505 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:27.581247 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: Sending DeleteTablet for 3 replicas of tablet 3b80d774b4a0413fb2fc528855b0b3e4
I20260928 09:04:27.581616 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b80d774b4a0413fb2fc528855b0b3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260928 09:04:27.581652 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b80d774b4a0413fb2fc528855b0b3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48076
I20260928 09:04:27.581627 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b80d774b4a0413fb2fc528855b0b3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42920
I20260928 09:04:27.581902 2038 tablet_replica.cc:330] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: stopping tablet replica
I20260928 09:04:27.582085 2038 raft_consensus.cc:2229] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:27.582120 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 09:04:27.582195 2038 raft_consensus.cc:2258] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 09:04:27.582257 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:27.582340 2037 tablet_replica.cc:330] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: stopping tablet replica
I20260928 09:04:27.582485 2037 raft_consensus.cc:2229] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:27.582617 2037 raft_consensus.cc:2258] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:27.582942 2038 ts_tablet_manager.cc:1905] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:27.583534 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55dc3f62534d0b9d90d0c7d7700f02 (DEFAULT_TABLE table=test-table [id=87bd8c8cd3d049d28d742d100aee9fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:27.583570 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55dc3f62534d0b9d90d0c7d7700f02 (DEFAULT_TABLE table=test-table [id=87bd8c8cd3d049d28d742d100aee9fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:27.583587 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55dc3f62534d0b9d90d0c7d7700f02 (DEFAULT_TABLE table=test-table [id=87bd8c8cd3d049d28d742d100aee9fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:27.583662 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c55dc3f62534d0b9d90d0c7d7700f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.583673 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c55dc3f62534d0b9d90d0c7d7700f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.583694 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c55dc3f62534d0b9d90d0c7d7700f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.584996 2015 tablet_bootstrap.cc:492] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Bootstrap starting.
I20260928 09:04:27.585145 2018 tablet_bootstrap.cc:492] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Bootstrap starting.
I20260928 09:04:27.585383 2038 ts_tablet_manager.cc:1918] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:27.585425 2015 tablet_bootstrap.cc:654] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.585446 2038 log.cc:1196] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/3b80d774b4a0413fb2fc528855b0b3e4
I20260928 09:04:27.585583 2018 tablet_bootstrap.cc:654] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.585736 2038 ts_tablet_manager.cc:1939] T 3b80d774b4a0413fb2fc528855b0b3e4 P 04b227592fdc4405ab87ff3fe5abea81: Deleting consensus metadata
I20260928 09:04:27.585786 2014 tablet_bootstrap.cc:492] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Bootstrap starting.
I20260928 09:04:27.586181 2014 tablet_bootstrap.cc:654] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.586177 1494 catalog_manager.cc:4841] TS 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317): tablet 3b80d774b4a0413fb2fc528855b0b3e4 (table test-table [id=45b44688d17d44f4bc5d40f9b91077a4]) successfully deleted
I20260928 09:04:27.586388 2037 ts_tablet_manager.cc:1905] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:27.586884 2014 tablet_bootstrap.cc:492] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: No bootstrap required, opened a new log
I20260928 09:04:27.586953 2014 ts_tablet_manager.cc:1397] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:27.587112 2014 raft_consensus.cc:348] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.587189 2014 raft_consensus.cc:374] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.587221 2014 raft_consensus.cc:729] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723d3c6aa9a948e8a46f34b62b5d23bd, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.587275 2014 consensus_queue.cc:260] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.587361 2014 ts_tablet_manager.cc:1428] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:27.587715 2037 ts_tablet_manager.cc:1918] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:27.587791 2037 log.cc:1196] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/3b80d774b4a0413fb2fc528855b0b3e4
I20260928 09:04:27.588073 2037 ts_tablet_manager.cc:1939] T 3b80d774b4a0413fb2fc528855b0b3e4 P 723d3c6aa9a948e8a46f34b62b5d23bd: Deleting consensus metadata
I20260928 09:04:27.588515 1493 catalog_manager.cc:4841] TS 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2:38283): tablet 3b80d774b4a0413fb2fc528855b0b3e4 (table test-table [id=45b44688d17d44f4bc5d40f9b91077a4]) successfully deleted
I20260928 09:04:27.588762 2018 tablet_bootstrap.cc:492] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: No bootstrap required, opened a new log
I20260928 09:04:27.588804 2018 ts_tablet_manager.cc:1397] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 09:04:27.588913 2018 raft_consensus.cc:348] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.588965 2018 raft_consensus.cc:374] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.588979 2018 raft_consensus.cc:729] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b227592fdc4405ab87ff3fe5abea81, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.589042 2018 consensus_queue.cc:260] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.589152 2018 ts_tablet_manager.cc:1428] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 09:04:27.589301 1992 tablet.cc:2369] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:27.589905 2039 tablet_replica.cc:330] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: stopping tablet replica
I20260928 09:04:27.589987 2015 tablet_bootstrap.cc:492] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: No bootstrap required, opened a new log
I20260928 09:04:27.590021 2015 ts_tablet_manager.cc:1397] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 09:04:27.590015 2039 raft_consensus.cc:2229] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:27.590256 2039 raft_consensus.cc:2258] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:27.590245 2015 raft_consensus.cc:348] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.590564 2015 raft_consensus.cc:374] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.590593 2015 raft_consensus.cc:729] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad8c4d55a04eddbdd62b0835d79e0c, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.590644 2015 consensus_queue.cc:260] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.590749 2015 ts_tablet_manager.cc:1428] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 09:04:27.590735 2039 ts_tablet_manager.cc:1905] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:27.592531 2039 ts_tablet_manager.cc:1918] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:27.592617 2039 log.cc:1196] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/3b80d774b4a0413fb2fc528855b0b3e4
I20260928 09:04:27.592835 2039 ts_tablet_manager.cc:1939] T 3b80d774b4a0413fb2fc528855b0b3e4 P 4cad8c4d55a04eddbdd62b0835d79e0c: Deleting consensus metadata
I20260928 09:04:27.593273 1495 catalog_manager.cc:4841] TS 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267): tablet 3b80d774b4a0413fb2fc528855b0b3e4 (table test-table [id=45b44688d17d44f4bc5d40f9b91077a4]) successfully deleted
I20260928 09:04:27.665756 2032 raft_consensus.cc:482] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:27.665951 2032 raft_consensus.cc:504] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.666170 2032 leader_election.cc:288] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267), 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.666363 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55dc3f62534d0b9d90d0c7d7700f02" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" is_pre_election: true
I20260928 09:04:27.666401 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55dc3f62534d0b9d90d0c7d7700f02" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b227592fdc4405ab87ff3fe5abea81" is_pre_election: true
I20260928 09:04:27.666469 1665 raft_consensus.cc:2454] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 0.
I20260928 09:04:27.666481 1945 raft_consensus.cc:2454] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 0.
I20260928 09:04:27.666643 1739 leader_election.cc:302] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cad8c4d55a04eddbdd62b0835d79e0c, 723d3c6aa9a948e8a46f34b62b5d23bd; no voters:
I20260928 09:04:27.666790 2032 raft_consensus.cc:2781] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:27.666858 2032 raft_consensus.cc:482] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:27.666879 2032 raft_consensus.cc:3037] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.667936 2032 raft_consensus.cc:504] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.668090 2032 leader_election.cc:288] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 election: Requested vote from peers 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267), 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.668259 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55dc3f62534d0b9d90d0c7d7700f02" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c"
I20260928 09:04:27.668277 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55dc3f62534d0b9d90d0c7d7700f02" candidate_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b227592fdc4405ab87ff3fe5abea81"
I20260928 09:04:27.668344 1665 raft_consensus.cc:3037] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.668381 1945 raft_consensus.cc:3037] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.669046 1945 raft_consensus.cc:2454] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 1.
I20260928 09:04:27.669142 1665 raft_consensus.cc:2454] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723d3c6aa9a948e8a46f34b62b5d23bd in term 1.
I20260928 09:04:27.669229 1738 leader_election.cc:302] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b227592fdc4405ab87ff3fe5abea81, 723d3c6aa9a948e8a46f34b62b5d23bd; no voters:
I20260928 09:04:27.669327 2032 raft_consensus.cc:2781] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:27.669366 2032 raft_consensus.cc:686] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 LEADER]: Becoming Leader. State: Replica: 723d3c6aa9a948e8a46f34b62b5d23bd, State: Running, Role: LEADER
I20260928 09:04:27.669426 2032 consensus_queue.cc:237] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.670084 1507 catalog_manager.cc:5495] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:27.709211 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.702260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42925 (local address 127.0.0.1:48022)
0928 09:04:27.702427 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.702439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.702529 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.702667 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.702668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.703050 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:27.703114 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.703116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.703762 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.703763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.704482 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.704484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.704808 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.704810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.704877 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.705416 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:27.705422 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:27.706944 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:27.708303 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.708309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.708311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.708506 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.708672 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:27.708673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:27.708673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:27.708726 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:27.708923 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:27.708928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:27.708972 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.709023 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.709116 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 09:04:27.709276 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.702340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.0.98.62:42925)
0928 09:04:27.702401 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.702403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.702518 (+ 115us) server_negotiation.cc:407] Connection header received
0928 09:04:27.702592 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.702593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.702601 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.702648 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.703194 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.703684 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.704562 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.704733 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.704783 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.707034 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.707035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.707035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.707042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.708231 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.708568 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.708569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.708569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.708616 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.708776 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.708776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.708776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.708870 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.708919 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.709053 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.709158 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":28}
I20260928 09:04:27.711146 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 09:04:27.711391 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:27.712846 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 3568cc2c9dd14bba849358c4fe3b5937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcd9e9a028a34bcda70b422b006657de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:27.712896 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 3568cc2c9dd14bba849358c4fe3b5937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcd9e9a028a34bcda70b422b006657de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:27.713008 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3568cc2c9dd14bba849358c4fe3b5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.713088 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3568cc2c9dd14bba849358c4fe3b5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.713690 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 3568cc2c9dd14bba849358c4fe3b5937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcd9e9a028a34bcda70b422b006657de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:27.713833 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3568cc2c9dd14bba849358c4fe3b5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:27.714521 2014 tablet_bootstrap.cc:492] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: Bootstrap starting.
I20260928 09:04:27.714521 2015 tablet_bootstrap.cc:492] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: Bootstrap starting.
I20260928 09:04:27.714839 2014 tablet_bootstrap.cc:654] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.714895 2015 tablet_bootstrap.cc:654] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.715444 2018 tablet_bootstrap.cc:492] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: Bootstrap starting.
I20260928 09:04:27.715731 2014 tablet_bootstrap.cc:492] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: No bootstrap required, opened a new log
I20260928 09:04:27.715772 2014 ts_tablet_manager.cc:1397] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:27.715832 2018 tablet_bootstrap.cc:654] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:27.715885 2014 raft_consensus.cc:348] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.715965 2014 raft_consensus.cc:374] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.715984 2014 raft_consensus.cc:729] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723d3c6aa9a948e8a46f34b62b5d23bd, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.716041 2014 consensus_queue.cc:260] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.716143 2014 ts_tablet_manager.cc:1428] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.716180 2032 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.716279 2032 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.716492 2018 tablet_bootstrap.cc:492] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: No bootstrap required, opened a new log
I20260928 09:04:27.716545 2018 ts_tablet_manager.cc:1397] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:27.716682 2018 raft_consensus.cc:348] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.716778 2018 raft_consensus.cc:374] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.716801 2018 raft_consensus.cc:729] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b227592fdc4405ab87ff3fe5abea81, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.716845 2018 consensus_queue.cc:260] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.716940 2018 ts_tablet_manager.cc:1428] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:27.717001 2022 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.717114 2022 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.717309 2015 tablet_bootstrap.cc:492] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: No bootstrap required, opened a new log
I20260928 09:04:27.717383 2015 ts_tablet_manager.cc:1397] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 09:04:27.717571 2015 raft_consensus.cc:348] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.717646 2015 raft_consensus.cc:374] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:27.717670 2015 raft_consensus.cc:729] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cad8c4d55a04eddbdd62b0835d79e0c, State: Initialized, Role: FOLLOWER
I20260928 09:04:27.717710 2015 consensus_queue.cc:260] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.717801 2015 ts_tablet_manager.cc:1428] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:27.717935 2019 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.718011 2019 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.806092 2019 raft_consensus.cc:482] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:27.806180 2019 raft_consensus.cc:504] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.806556 2019 leader_election.cc:288] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2:38283), 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.810724 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.806761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54213 (local address 127.0.98.2:38283)
0928 09:04:27.806808 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.806810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.806939 (+ 129us) server_negotiation.cc:407] Connection header received
0928 09:04:27.807038 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.807040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.807050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.807110 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.807325 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.808086 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.809812 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.810350 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.810423 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.810433 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.810614 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.810649 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:04:27.810722 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.806813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45081 (local address 127.0.98.3:37317)
0928 09:04:27.806867 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.806869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.806994 (+ 125us) server_negotiation.cc:407] Connection header received
0928 09:04:27.807041 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.807042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.807054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.807113 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:27.807325 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.808070 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.809811 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.810345 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.810423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.810433 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.810620 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.810654 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 09:04:27.810869 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.806721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.98.1:45081)
0928 09:04:27.806873 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.806931 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.806985 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.807136 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.807137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.807138 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.807246 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.807249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.808193 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.808194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.809712 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.809715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.810441 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.810442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.810534 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.810539 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.810607 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.810660 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":127,"thread_start_us":96,"threads_started":1}
I20260928 09:04:27.811090 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.806705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.98.1:54213)
0928 09:04:27.806872 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.806888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.806964 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.807136 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.807137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.807138 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:27.807247 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.807250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.808192 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.808194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.809712 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.809715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.810439 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.810441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.810534 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.810541 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.810602 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.810660 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":104,"thread_start_us":87,"threads_started":1}
I20260928 09:04:27.811175 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3568cc2c9dd14bba849358c4fe3b5937" candidate_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b227592fdc4405ab87ff3fe5abea81" is_pre_election: true
I20260928 09:04:27.811282 1945 raft_consensus.cc:2454] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad8c4d55a04eddbdd62b0835d79e0c in term 0.
I20260928 09:04:27.811338 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3568cc2c9dd14bba849358c4fe3b5937" candidate_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" is_pre_election: true
I20260928 09:04:27.811465 1598 leader_election.cc:302] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b227592fdc4405ab87ff3fe5abea81, 4cad8c4d55a04eddbdd62b0835d79e0c; no voters:
I20260928 09:04:27.811520 1805 raft_consensus.cc:2454] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cad8c4d55a04eddbdd62b0835d79e0c in term 0.
I20260928 09:04:27.811614 2019 raft_consensus.cc:2781] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:27.811668 2019 raft_consensus.cc:482] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:27.811697 2019 raft_consensus.cc:3037] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.812386 2019 raft_consensus.cc:504] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.812536 2019 leader_election.cc:288] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [CANDIDATE]: Term 1 election: Requested vote from peers 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2:38283), 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317)
I20260928 09:04:27.812666 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3568cc2c9dd14bba849358c4fe3b5937" candidate_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd"
I20260928 09:04:27.812698 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3568cc2c9dd14bba849358c4fe3b5937" candidate_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b227592fdc4405ab87ff3fe5abea81"
I20260928 09:04:27.812758 1805 raft_consensus.cc:3037] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.812767 1945 raft_consensus.cc:3037] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:27.813484 1945 raft_consensus.cc:2454] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad8c4d55a04eddbdd62b0835d79e0c in term 1.
I20260928 09:04:27.813515 1805 raft_consensus.cc:2454] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cad8c4d55a04eddbdd62b0835d79e0c in term 1.
I20260928 09:04:27.813633 1598 leader_election.cc:302] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b227592fdc4405ab87ff3fe5abea81, 4cad8c4d55a04eddbdd62b0835d79e0c; no voters:
I20260928 09:04:27.813776 2019 raft_consensus.cc:2781] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:27.813912 2019 raft_consensus.cc:686] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 LEADER]: Becoming Leader. State: Replica: 4cad8c4d55a04eddbdd62b0835d79e0c, State: Running, Role: LEADER
I20260928 09:04:27.813964 2019 consensus_queue.cc:237] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260928 09:04:27.814370 2055 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: TxnStatusTablet state changed. Reason: New leader 4cad8c4d55a04eddbdd62b0835d79e0c. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.814500 2055 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:27.814680 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 09:04:27.814802 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 09:04:27.814774 1507 catalog_manager.cc:5495] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:27.834263 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.831621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46267 (local address 127.0.0.1:42930)
0928 09:04:27.831695 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.831707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.831757 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.831956 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.831957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.831958 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:27.832010 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.832012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.832669 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.832670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.833271 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.833273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.833718 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.833719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.833787 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.833803 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.834082 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.834091 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.834146 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.834198 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 09:04:27.834251 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.831693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.0.98.1:46267)
0928 09:04:27.831739 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.831741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.831751 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:27.831843 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.831844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.831867 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.831945 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:27.832099 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.832584 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.833378 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.833636 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.833717 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.833904 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.834012 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.834094 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.834151 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.834174 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:04:27.835412 1945 raft_consensus.cc:1261] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Refusing update from remote peer 4cad8c4d55a04eddbdd62b0835d79e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:27.835522 1805 raft_consensus.cc:1261] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Refusing update from remote peer 4cad8c4d55a04eddbdd62b0835d79e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:27.835834 2055 consensus_queue.cc:1035] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:27.835973 2019 consensus_queue.cc:1035] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:27.836601 2022 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: TxnStatusTablet state changed. Reason: New leader 4cad8c4d55a04eddbdd62b0835d79e0c. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.836701 2022 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.836774 2032 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: TxnStatusTablet state changed. Reason: New leader 4cad8c4d55a04eddbdd62b0835d79e0c. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.836844 2032 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.837635 2062 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.837620 2019 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.837728 2019 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:27.837803 2062 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 4cad8c4d55a04eddbdd62b0835d79e0c: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:27.838146 2022 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.838227 2022 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 04b227592fdc4405ab87ff3fe5abea81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.838816 2032 tablet_replica.cc:439] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 } } peers { permanent_uuid: "723d3c6aa9a948e8a46f34b62b5d23bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260928 09:04:27.838896 2032 tablet_replica.cc:442] T 3568cc2c9dd14bba849358c4fe3b5937 P 723d3c6aa9a948e8a46f34b62b5d23bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:27.843976 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.841360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39583 (local address 127.0.98.1:46267)
0928 09:04:27.841415 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.841418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.841462 (+ 44us) server_negotiation.cc:407] Connection header received
0928 09:04:27.841527 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.841528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.841578 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.841651 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:27.841763 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.842346 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.843260 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.843461 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.843543 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.843729 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.843769 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.843829 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.843894 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.843911 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:04:27.844010 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.841236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46267 (local address 127.0.98.2:39583)
0928 09:04:27.841406 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.841420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.841478 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.841654 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.841655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.841656 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:27.841703 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.841705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.842453 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.842454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.843154 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.843156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.843545 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.843546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.843600 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.843635 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.843830 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.843833 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.843885 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.843933 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260928 09:04:27.845825 1944 raft_consensus.cc:1261] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Refusing update from remote peer 723d3c6aa9a948e8a46f34b62b5d23bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:27.845865 1665 raft_consensus.cc:1261] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Refusing update from remote peer 723d3c6aa9a948e8a46f34b62b5d23bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:27.846017 2021 consensus_queue.cc:1035] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b227592fdc4405ab87ff3fe5abea81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:27.846101 2032 consensus_queue.cc:1035] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cad8c4d55a04eddbdd62b0835d79e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:27.854373 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.851623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45525 (local address 127.0.98.2:38283)
0928 09:04:27.851672 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.851674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.851737 (+ 63us) server_negotiation.cc:407] Connection header received
0928 09:04:27.851798 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.851798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.851808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.851854 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:27.851981 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.852533 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.853717 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.853904 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.853954 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.854156 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.854189 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.854241 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.854300 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.854317 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 09:04:27.854439 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.851445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.98.1:45525)
0928 09:04:27.851670 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:27.851682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:27.851742 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:27.851857 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:27.851857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:27.851859 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:27.851925 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.851927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.852611 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.852612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.853628 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:27.853631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.853974 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.853975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:27.854039 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.854056 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:27.854246 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:27.854247 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:27.854298 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:04:27.854341 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260928 09:04:27.862500 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:27.862608 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:27.863253 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b5f1ba834241c58169e20388e504ed: Sending DeleteTablet for 3 replicas of tablet 0c55dc3f62534d0b9d90d0c7d7700f02
I20260928 09:04:27.863507 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55dc3f62534d0b9d90d0c7d7700f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42920
I20260928 09:04:27.863521 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55dc3f62534d0b9d90d0c7d7700f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260928 09:04:27.863605 2038 tablet_replica.cc:330] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: stopping tablet replica
I20260928 09:04:27.863605 2039 tablet_replica.cc:330] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: stopping tablet replica
I20260928 09:04:27.863651 2039 raft_consensus.cc:2229] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:27.863652 2038 raft_consensus.cc:2229] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:27.863680 2039 raft_consensus.cc:2258] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:27.863686 2038 raft_consensus.cc:2258] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:27.864012 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55dc3f62534d0b9d90d0c7d7700f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48076
I20260928 09:04:27.864033 2039 ts_tablet_manager.cc:1905] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:27.864274 2038 ts_tablet_manager.cc:1905] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:27.865345 2037 tablet_replica.cc:330] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: stopping tablet replica
I20260928 09:04:27.865440 2037 raft_consensus.cc:2229] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:27.865499 2037 raft_consensus.cc:2258] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:27.865805 2037 ts_tablet_manager.cc:1905] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:27.865828 2039 ts_tablet_manager.cc:1918] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:27.865880 2038 ts_tablet_manager.cc:1918] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:27.865903 2039 log.cc:1196] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/0c55dc3f62534d0b9d90d0c7d7700f02
I20260928 09:04:27.865931 2038 log.cc:1196] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/0c55dc3f62534d0b9d90d0c7d7700f02
I20260928 09:04:27.866122 2039 ts_tablet_manager.cc:1939] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 4cad8c4d55a04eddbdd62b0835d79e0c: Deleting consensus metadata
I20260928 09:04:27.866150 2038 ts_tablet_manager.cc:1939] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 04b227592fdc4405ab87ff3fe5abea81: Deleting consensus metadata
I20260928 09:04:27.866468 1495 catalog_manager.cc:4841] TS 4cad8c4d55a04eddbdd62b0835d79e0c (127.0.98.1:46267): tablet 0c55dc3f62534d0b9d90d0c7d7700f02 (table test-table [id=87bd8c8cd3d049d28d742d100aee9fbd]) successfully deleted
I20260928 09:04:27.866576 1494 catalog_manager.cc:4841] TS 04b227592fdc4405ab87ff3fe5abea81 (127.0.98.3:37317): tablet 0c55dc3f62534d0b9d90d0c7d7700f02 (table test-table [id=87bd8c8cd3d049d28d742d100aee9fbd]) successfully deleted
I20260928 09:04:27.867421 2037 ts_tablet_manager.cc:1918] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:27.867458 2037 log.cc:1196] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/0c55dc3f62534d0b9d90d0c7d7700f02
I20260928 09:04:27.867604 2037 ts_tablet_manager.cc:1939] T 0c55dc3f62534d0b9d90d0c7d7700f02 P 723d3c6aa9a948e8a46f34b62b5d23bd: Deleting consensus metadata
I20260928 09:04:27.868036 1493 catalog_manager.cc:4841] TS 723d3c6aa9a948e8a46f34b62b5d23bd (127.0.98.2:38283): tablet 0c55dc3f62534d0b9d90d0c7d7700f02 (table test-table [id=87bd8c8cd3d049d28d742d100aee9fbd]) successfully deleted
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:27.875960 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.866664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.98.1:46267)
0928 09:04:27.866706 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.866708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.866873 (+ 165us) server_negotiation.cc:407] Connection header received
0928 09:04:27.866924 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.866925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.866935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.866986 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.868085 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.868569 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.869629 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.869813 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.869863 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.873160 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.873169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.873169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.873177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.874786 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.875163 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.875167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.875168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.875229 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.875430 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.875431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.875431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.875565 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.875727 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.875774 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.875847 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:04:27.876230 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42946
I20260928 09:04:27.883616 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:27.877155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.0.98.62:42925)
0928 09:04:27.877209 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:27.877211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:27.877220 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:27.877284 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:27.877285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:27.877296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:27.877340 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:27.877923 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.878743 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.879600 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:27.879891 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:27.879966 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:27.881122 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:27.881124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:27.881124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:27.881132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:27.882610 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.882915 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.882916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.882916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.882978 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:27.883152 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:27.883153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:27.883153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:27.883279 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:27.883376 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:27.883444 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 09:04:27.883523 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 09:04:27.883798 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48028
I20260928 09:04:27.884372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1580
I20260928 09:04:27.889796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1720
I20260928 09:04:27.895285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1860
I20260928 09:04:27.899829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 1476
2026-09-28T09:04:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2366 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[2077](info): set up 2 sockets
Sep 28 09:04:27 dist-test-slave-smns krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260928 09:04:28.941294 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:29.067823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:29.150759 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:29.151000 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:29.151036 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:29.152180 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:29.152247 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 09:04:29.152273 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:29.152297 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:29.152328 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:29.152359 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:29.153899 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46151
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:29.154173 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:29.154409 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:29.159210 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.159224 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.159246 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:29.159447 2090 server_base.cc:1034] running on GCE node
I20260928 09:04:29.159698 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:29.160005 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:29.161144 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790586269161125 us; error 37 us; skew 500 ppm
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:29.163540 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:29.164000 2090 webserver.cc:460] Webserver started at http://127.0.98.62:46255/ using document root <none> and password file <none>
I20260928 09:04:29.164234 2090 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:29.164301 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:29.164475 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:29.165534 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "f41c4d16ce684330b551d3ca972dc05d"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "a54aad413e9303125b7f34dab9f2af81"
server_key_iv: "faf9206dc001c5651da961ab81adde0e"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.165980 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "f41c4d16ce684330b551d3ca972dc05d"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "a54aad413e9303125b7f34dab9f2af81"
server_key_iv: "faf9206dc001c5651da961ab81adde0e"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.167331 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:29.168082 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.168246 2090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:29.168346 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "f41c4d16ce684330b551d3ca972dc05d"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "a54aad413e9303125b7f34dab9f2af81"
server_key_iv: "faf9206dc001c5651da961ab81adde0e"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.168427 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:29.182510 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:29.184562 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:29.184736 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:29.189577 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46151
I20260928 09:04:29.189631 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46151 every 8 connection(s)
I20260928 09:04:29.189957 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:29.190616 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:29.192719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 2090
I20260928 09:04:29.192796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:29.192973 392 external_mini_cluster.cc:1428] Setting key 8f60876b14b9293871551ef093d885ab
I20260928 09:04:29.193042 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Bootstrap starting.
I20260928 09:04:29.193850 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:29.194164 2168 log.cc:824] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:29.194890 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: No bootstrap required, opened a new log
I20260928 09:04:29.197238 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41c4d16ce684330b551d3ca972dc05d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46151 } }
I20260928 09:04:29.197382 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:29.197414 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41c4d16ce684330b551d3ca972dc05d, State: Initialized, Role: FOLLOWER
I20260928 09:04:29.197597 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41c4d16ce684330b551d3ca972dc05d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46151 } }
I20260928 09:04:29.197671 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:29.197715 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:29.197762 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:29.198282 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41c4d16ce684330b551d3ca972dc05d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46151 } }
I20260928 09:04:29.198412 2168 leader_election.cc:302] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41c4d16ce684330b551d3ca972dc05d; no voters:
I20260928 09:04:29.198581 2168 leader_election.cc:288] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:29.198622 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:29.198747 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [term 1 LEADER]: Becoming Leader. State: Replica: f41c4d16ce684330b551d3ca972dc05d, State: Running, Role: LEADER
I20260928 09:04:29.198846 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:29.198846 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41c4d16ce684330b551d3ca972dc05d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46151 } }
I20260928 09:04:29.199337 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41c4d16ce684330b551d3ca972dc05d. Latest consensus state: current_term: 1 leader_uuid: "f41c4d16ce684330b551d3ca972dc05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41c4d16ce684330b551d3ca972dc05d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46151 } } }
I20260928 09:04:29.199509 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:29.199935 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:29.200480 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:29.200632 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41c4d16ce684330b551d3ca972dc05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41c4d16ce684330b551d3ca972dc05d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46151 } } }
I20260928 09:04:29.200951 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:29.202478 2182 catalog_manager.cc:1347] Generated new cluster ID: 8a8303afd4594af1a7c621c7fcfa2104
I20260928 09:04:29.202530 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:29.204643 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.193528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.0.98.62:46151)
0928 09:04:29.193752 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.193761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.193793 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:04:29.194447 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.194456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.194522 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.194679 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.195322 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.196790 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.197646 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.198172 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.198296 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.201201 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.201206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.201206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.201219 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.203105 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.203482 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.203485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.203485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.203575 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.203785 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.203786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.203787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.204025 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.204202 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.204305 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.204439 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260928 09:04:29.234966 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:29.235510 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:29.238039 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Generated new TSK 0
I20260928 09:04:29.238233 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:29.266486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46151
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:29.352818 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:29.353031 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:29.353060 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:29.354167 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:29.354206 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:29.354228 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 09:04:29.354238 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:29.354255 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:29.355638 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:29.355903 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:29.356144 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:29.361123 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.361117 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.361120 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:29.361349 2194 server_base.cc:1034] running on GCE node
I20260928 09:04:29.361533 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:29.361819 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:29.362965 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790586269362954 us; error 31 us; skew 500 ppm
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:29.365411 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:29.365967 2194 webserver.cc:460] Webserver started at http://127.0.98.1:45917/ using document root <none> and password file <none>
I20260928 09:04:29.366153 2194 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:29.366196 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:29.366283 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:29.367432 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "47e97f22e1cd407b9c5b851438b60aa9"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "d1815033189c29a349ef2ac947d23a73"
server_key_iv: "c41ffa6e3cd529e17c5d842089bf6d41"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.367733 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47e97f22e1cd407b9c5b851438b60aa9"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "d1815033189c29a349ef2ac947d23a73"
server_key_iv: "c41ffa6e3cd529e17c5d842089bf6d41"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.369050 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:04:29.369809 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.370031 2194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:29.370118 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "47e97f22e1cd407b9c5b851438b60aa9"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "d1815033189c29a349ef2ac947d23a73"
server_key_iv: "c41ffa6e3cd529e17c5d842089bf6d41"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.370177 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:29.378486 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:29.380455 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:29.380565 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:29.381052 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:29.381093 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.381114 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:29.381127 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.388136 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38391
I20260928 09:04:29.388257 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38391 every 8 connection(s)
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:29.388579 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:29.391566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 2194
I20260928 09:04:29.391664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:29.391785 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.381519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58717 (local address 127.0.98.62:46151)
0928 09:04:29.381634 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.381636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.381888 (+ 252us) server_negotiation.cc:407] Connection header received
0928 09:04:29.382378 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.382379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.382393 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.382481 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.383283 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.384144 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.385796 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.386169 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.386249 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.389162 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.389163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.389163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.389171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.390473 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.390838 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.390839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.390839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.390902 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.391128 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.391129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.391129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.391233 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.391285 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.391594 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.391690 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 09:04:29.391836 392 external_mini_cluster.cc:1428] Setting key fbab7a1932b6038963c500e36df81059
I20260928 09:04:29.391892 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.381502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.98.1:58717)
0928 09:04:29.381702 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.381743 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.382218 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.382534 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.382537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.382860 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:29.383158 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.383165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.384294 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.384295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.385684 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.385686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.386280 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.386282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.386404 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.387002 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:29.387024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:29.389046 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:29.390548 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.390553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.390555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.390769 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.390969 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.390970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.390970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.391072 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.391315 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:29.391321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:29.391439 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.391537 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.391671 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":133,"threads_started":1}
I20260928 09:04:29.392539 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46151
I20260928 09:04:29.392620 2325 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:29.392812 2325 heartbeater.cc:511] Master 127.0.98.62:46151 requested a full tablet report, sending...
I20260928 09:04:29.393332 2122 ts_manager.cc:194] Registered new tserver with Master: 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391)
I20260928 09:04:29.394251 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58717
I20260928 09:04:29.400028 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.393532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.98.1:44915)
0928 09:04:29.393697 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.393706 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.393800 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.393963 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.393964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.394184 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:29.394264 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.394266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.395139 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.395140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.395661 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.395662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.396133 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.396134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.396197 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.396640 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:29.396649 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:29.397001 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:29.398904 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.398905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.398905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.399069 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.399284 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.399284 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.399285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.399335 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.399629 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:29.399630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:29.399668 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.399872 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.399957 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260928 09:04:29.400104 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.393599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44915 (local address 127.0.98.62:46151)
0928 09:04:29.393640 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.393641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.393776 (+ 135us) server_negotiation.cc:407] Connection header received
0928 09:04:29.393859 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.393860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.393870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.393918 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.394390 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.395037 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.395748 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.396054 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.396124 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.397085 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.397087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.397087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.397095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.398805 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.399153 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.399154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.399155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.399213 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.399402 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.399403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.399404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.399551 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.399624 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.399908 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.399988 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:29.420872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46151
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:29.502163 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:29.502406 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:29.502447 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:29.503597 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:29.503669 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:29.503706 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 09:04:29.503746 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:29.503789 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:29.505290 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:29.505555 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:29.505829 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:29.510962 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.510977 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.510977 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:29.511111 2334 server_base.cc:1034] running on GCE node
I20260928 09:04:29.511294 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:29.511548 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:29.512710 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790586269512699 us; error 39 us; skew 500 ppm
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:29.515210 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:29.515662 2334 webserver.cc:460] Webserver started at http://127.0.98.2:35047/ using document root <none> and password file <none>
I20260928 09:04:29.515885 2334 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:29.515951 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:29.516100 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:29.517190 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "78c1e7620b5832ae00ffc6d21e031517"
server_key_iv: "29584e54e3df4d17cb21d6fbbe7e7a8c"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.517606 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "78c1e7620b5832ae00ffc6d21e031517"
server_key_iv: "29584e54e3df4d17cb21d6fbbe7e7a8c"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.518949 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:29.519654 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.519876 2334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:29.519976 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "78c1e7620b5832ae00ffc6d21e031517"
server_key_iv: "29584e54e3df4d17cb21d6fbbe7e7a8c"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.520061 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:29.533865 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:29.536180 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:29.536294 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:29.536829 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:29.536895 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.536928 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:29.536945 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:29.544140 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43903
I20260928 09:04:29.544471 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43903 every 8 connection(s)
I20260928 09:04:29.544608 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:29.545534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 2334
I20260928 09:04:29.545666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:29.545846 392 external_mini_cluster.cc:1428] Setting key 52ebcd48217218842ad5ecf834293f3d
I20260928 09:04:29.547087 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.537182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35821 (local address 127.0.98.62:46151)
0928 09:04:29.537243 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.537245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.537531 (+ 286us) server_negotiation.cc:407] Connection header received
0928 09:04:29.538095 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.538097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.538108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.538168 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.538855 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.539363 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.540359 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.540672 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.540744 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.543850 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.543852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.543852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.543889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.545500 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.546034 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.546036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.546036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.546101 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.546356 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.546357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.546357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.546515 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.546587 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.546950 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.547021 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:04:29.547398 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.537181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.98.2:35821)
0928 09:04:29.537380 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.537437 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.537981 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.538247 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.538251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.538566 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:29.538760 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.538765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.539457 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.539458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.540248 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.540249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.540751 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.540752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.540922 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.541437 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:29.541445 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:29.543707 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:29.545644 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.545651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.545653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.545968 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.546177 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.546178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.546178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.546284 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.546591 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:29.546598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:29.546792 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.546919 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.547107 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":124,"threads_started":1}
I20260928 09:04:29.548297 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46151
I20260928 09:04:29.548424 2465 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:29.548638 2465 heartbeater.cc:511] Master 127.0.98.62:46151 requested a full tablet report, sending...
I20260928 09:04:29.549073 2120 ts_manager.cc:194] Registered new tserver with Master: 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903)
I20260928 09:04:29.549659 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35821
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:04:29.555327 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.549395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51395 (local address 127.0.98.62:46151)
0928 09:04:29.549431 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.549432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.549763 (+ 331us) server_negotiation.cc:407] Connection header received
0928 09:04:29.549836 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.549837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.549848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.549910 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.550277 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.550754 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.551472 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.551675 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.551759 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.552736 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.552737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.552738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.552744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.554151 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.554537 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.554538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.554539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.554594 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.554786 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.554787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.554787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.554934 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.555014 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.555164 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.555245 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:29.555437 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.549333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.98.2:51395)
0928 09:04:29.549684 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.549696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.549774 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.549915 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.549916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.550114 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:29.550195 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.550197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.550849 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.550850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.551381 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.551382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.551761 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.551762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.551843 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.552360 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:29.552364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:29.552646 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:29.554250 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.554251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.554251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.554454 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.554659 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.554660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.554660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.554717 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.555006 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:29.555007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:29.555051 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.555138 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.555241 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:29.572705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46151
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:29.665885 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:29.666114 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:29.666139 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:29.667352 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:29.667394 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:29.667415 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 09:04:29.667428 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:29.667444 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:29.668860 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:29.669064 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:29.669246 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:29.674357 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.674357 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:29.674335 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:29.674618 2474 server_base.cc:1034] running on GCE node
I20260928 09:04:29.674863 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:29.675148 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:29.676314 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790586269676303 us; error 30 us; skew 500 ppm
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:29.678957 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:29.679394 2474 webserver.cc:460] Webserver started at http://127.0.98.3:41283/ using document root <none> and password file <none>
I20260928 09:04:29.679641 2474 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:29.679713 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:29.679863 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:29.680938 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "da659d13c3a4452e9efcf1c98b3f95be"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "be68b68315d7252a3620aa8813017306"
server_key_iv: "5dfd4ba2ff8bb2a7f0dca8bbae802b5a"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.681300 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da659d13c3a4452e9efcf1c98b3f95be"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "be68b68315d7252a3620aa8813017306"
server_key_iv: "5dfd4ba2ff8bb2a7f0dca8bbae802b5a"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.682722 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:04:29.683506 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.683686 2474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:29.683782 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "da659d13c3a4452e9efcf1c98b3f95be"
format_stamp: "Formatted at 2026-09-28 09:04:29 on dist-test-slave-smns"
server_key: "be68b68315d7252a3620aa8813017306"
server_key_iv: "5dfd4ba2ff8bb2a7f0dca8bbae802b5a"
server_key_version: "encryptionkey@0"
I20260928 09:04:29.683867 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:29.702764 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:29.705159 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:29.705354 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:29.706095 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:29.706187 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.706257 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:29.706305 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:29.713310 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33699
I20260928 09:04:29.713387 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33699 every 8 connection(s)
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:29.713737 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:29.716596 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.706850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52675 (local address 127.0.98.62:46151)
0928 09:04:29.706901 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.706903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.706913 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:29.707529 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.707531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.707543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.707612 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.708802 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.709665 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.711202 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.711562 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.711646 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.714284 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.714286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.714286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.714292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.715436 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.715796 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.715796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.715797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.715846 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.716111 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.716112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.716112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.716216 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.716267 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.716471 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.716531 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 09:04:29.716769 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.706502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.98.3:52675)
0928 09:04:29.706692 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.706743 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.707392 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.707718 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.707722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.708035 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:29.708306 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.708312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.709764 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.709766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.711096 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.711099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.711654 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.711655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.711759 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.712307 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:29.712320 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:29.714181 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:29.715503 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.715509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.715510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.715725 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.715940 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.715940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.715941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.716050 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.716270 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:29.716274 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:29.716383 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.716461 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.716576 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260928 09:04:29.717272 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46151
I20260928 09:04:29.717356 2605 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:29.717577 2605 heartbeater.cc:511] Master 127.0.98.62:46151 requested a full tablet report, sending...
I20260928 09:04:29.717953 2122 ts_manager.cc:194] Registered new tserver with Master: da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:29.718382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 2474
I20260928 09:04:29.718401 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52675
I20260928 09:04:29.718474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:29.718670 392 external_mini_cluster.cc:1428] Setting key 94429ca93ffd0f001c0a80a2392b592c
I20260928 09:04:29.719436 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:04:29.725322 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.718418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50117 (local address 127.0.98.62:46151)
0928 09:04:29.718454 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.718456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.718564 (+ 108us) server_negotiation.cc:407] Connection header received
0928 09:04:29.718626 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.718627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.718636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.718689 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.719580 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.720035 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.721164 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.721428 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.721491 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.722590 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.722592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.722592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.722600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.724199 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.724561 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.724563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.724564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.724621 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.724813 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.724814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.724814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.724927 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.725002 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.725151 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.725225 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:29.725334 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.718353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.98.3:50117)
0928 09:04:29.718506 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.718517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.718577 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.719217 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.719217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.719444 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:29.719517 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.719520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.720140 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.720141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.721042 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.721045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.721493 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.721494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.721597 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.722040 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:29.722046 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:29.722463 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:29.724296 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.724297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.724298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.724474 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.724687 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:29.724688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:29.724688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:29.724749 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:29.724994 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:29.724995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:29.725037 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.725123 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.725228 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 09:04:29 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:29.743477 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.736544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.0.98.62:46151)
0928 09:04:29.736603 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.736605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.736615 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:29.736644 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.736644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.736653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.736703 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:29.737234 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.737791 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.738625 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.738807 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.738864 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.740947 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:29.740948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:29.740949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:29.740956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:29.742413 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.742723 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.742724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.742724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.742773 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:29.743014 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:29.743015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:29.743015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:29.743110 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:29.743179 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.743306 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.743394 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:04:29.745507 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:29.747090 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:29.753708 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.749921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.0.98.3:33699)
0928 09:04:29.750109 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.750113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.750121 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:29.750159 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.750162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.750177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.750260 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:29.750506 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.751575 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.752694 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.753212 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.753318 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.753384 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.753574 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.753617 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260928 09:04:29.753698 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.749673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33699 (local address 127.0.0.1:56596)
0928 09:04:29.749977 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.749985 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.750023 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.750327 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.750327 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.750330 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:29.750401 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.750402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.751662 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.751663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.752590 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.752592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.753321 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.753322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.753404 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.753417 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.753526 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.753606 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":22,"thread_start_us":76,"threads_started":1}
I20260928 09:04:29.753809 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.749414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38391 (local address 127.0.0.1:49734)
0928 09:04:29.749782 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.749792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.749861 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.750167 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.750170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.750172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:29.750238 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.750240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.751616 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.751617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.752660 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.752662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.753593 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.753594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.753651 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.753655 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.753698 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.753738 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":66,"threads_started":1}
I20260928 09:04:29.754101 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.749504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.98.1:38391)
0928 09:04:29.749731 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.749736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.749950 (+ 214us) server_negotiation.cc:407] Connection header received
0928 09:04:29.749999 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.750002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.750020 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.750090 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:29.750322 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.751527 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.752772 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.753476 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.753533 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.753939 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.753967 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.754008 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":127,"threads_started":1}
I20260928 09:04:29.754606 2539 tablet_service.cc:1467] Processing CreateTablet for tablet acabe40ea32c49918f5fc0d4ebeefe1b (DEFAULT_TABLE table=test-table [id=b023392ee56c4b768f1b28eaae134c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:29.754902 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acabe40ea32c49918f5fc0d4ebeefe1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:29.754881 2259 tablet_service.cc:1467] Processing CreateTablet for tablet acabe40ea32c49918f5fc0d4ebeefe1b (DEFAULT_TABLE table=test-table [id=b023392ee56c4b768f1b28eaae134c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:29.755242 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acabe40ea32c49918f5fc0d4ebeefe1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:29.757639 2628 tablet_bootstrap.cc:492] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Bootstrap starting.
I20260928 09:04:29.758615 2628 tablet_bootstrap.cc:654] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:29.758941 2628 log.cc:824] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:29.759784 2628 tablet_bootstrap.cc:492] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: No bootstrap required, opened a new log
I20260928 09:04:29.759872 2628 ts_tablet_manager.cc:1397] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:04:29.760228 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.749408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.0.1:51270)
0928 09:04:29.749739 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.749786 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.749861 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.750327 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.750327 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.750331 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:29.750409 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.750411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.751328 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.751329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.752606 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.752608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.759973 (+ 7365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.759974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.760078 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.760084 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.760131 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.760170 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":73,"threads_started":1}
I20260928 09:04:29.760627 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.749607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.98.2:43903)
0928 09:04:29.749936 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.749941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.749952 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:29.750174 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.750177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.750194 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.750264 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:29.750532 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.751229 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.752725 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.753457 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.753531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.760452 (+ 6921us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.760483 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.760534 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":78,"threads_started":1}
I20260928 09:04:29.761317 2399 tablet_service.cc:1467] Processing CreateTablet for tablet acabe40ea32c49918f5fc0d4ebeefe1b (DEFAULT_TABLE table=test-table [id=b023392ee56c4b768f1b28eaae134c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:29.761749 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acabe40ea32c49918f5fc0d4ebeefe1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:29.761732 2630 tablet_bootstrap.cc:492] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Bootstrap starting.
I20260928 09:04:29.761730 2628 raft_consensus.cc:348] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.761912 2628 raft_consensus.cc:374] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:29.761986 2628 raft_consensus.cc:729] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da659d13c3a4452e9efcf1c98b3f95be, State: Initialized, Role: FOLLOWER
I20260928 09:04:29.762118 2628 consensus_queue.cc:260] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.762362 2630 tablet_bootstrap.cc:654] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:29.762428 2605 heartbeater.cc:503] Master 127.0.98.62:46151 was elected leader, sending a full tablet report...
I20260928 09:04:29.762428 2628 ts_tablet_manager.cc:1428] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:04:29.762768 2630 log.cc:824] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:29.763640 2630 tablet_bootstrap.cc:492] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: No bootstrap required, opened a new log
I20260928 09:04:29.763715 2630 ts_tablet_manager.cc:1397] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:29.765513 2633 tablet_bootstrap.cc:492] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Bootstrap starting.
I20260928 09:04:29.766095 2633 tablet_bootstrap.cc:654] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:29.766005 2630 raft_consensus.cc:348] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.766170 2630 raft_consensus.cc:374] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:29.766219 2630 raft_consensus.cc:729] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Initialized, Role: FOLLOWER
I20260928 09:04:29.766350 2630 consensus_queue.cc:260] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.766431 2633 log.cc:824] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:29.766637 2630 ts_tablet_manager.cc:1428] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:04:29.766676 2325 heartbeater.cc:503] Master 127.0.98.62:46151 was elected leader, sending a full tablet report...
I20260928 09:04:29.767261 2633 tablet_bootstrap.cc:492] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: No bootstrap required, opened a new log
I20260928 09:04:29.767342 2633 ts_tablet_manager.cc:1397] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:29.769003 2633 raft_consensus.cc:348] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.769124 2633 raft_consensus.cc:374] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:29.769145 2633 raft_consensus.cc:729] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc9ed8f250f430da9ae99ed24d14c57, State: Initialized, Role: FOLLOWER
I20260928 09:04:29.769281 2633 consensus_queue.cc:260] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.769536 2633 ts_tablet_manager.cc:1428] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:29.769601 2465 heartbeater.cc:503] Master 127.0.98.62:46151 was elected leader, sending a full tablet report...
W20260928 09:04:29.795879 2466 tablet.cc:2369] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:04:29.889456 2326 tablet.cc:2369] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:04:29.964447 2606 tablet.cc:2369] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:29.973131 2634 raft_consensus.cc:482] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:29.973278 2634 raft_consensus.cc:504] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:29.973721 2634 leader_election.cc:288] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:29.976857 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.973970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51675 (local address 127.0.98.2:43903)
0928 09:04:29.974045 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.974047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.974168 (+ 121us) server_negotiation.cc:407] Connection header received
0928 09:04:29.974224 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.974225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.974238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.974298 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:29.974516 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.975076 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.976115 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.976505 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.976561 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.976570 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.976740 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.976785 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:04:29.976853 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.973890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.98.1:51675)
0928 09:04:29.974059 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.974071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.974142 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.974336 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.974337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.974341 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:29.974435 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.974437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.975165 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.975166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.976011 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.976013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.976588 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.976589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.976660 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.976664 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.976718 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.976750 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":2,"thread_start_us":123,"threads_started":1}
I20260928 09:04:29.976956 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.974270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53543 (local address 127.0.98.3:33699)
0928 09:04:29.974340 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:29.974341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:29.974350 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:29.974390 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:29.974391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:29.974401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:29.974459 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:29.974610 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.975205 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.976141 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.976564 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.976634 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.976644 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:29.976829 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 09:04:29.976871 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 09:04:29.977128 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:29.974165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33699 (local address 127.0.98.1:53543)
0928 09:04:29.974273 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:29.974284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:29.974330 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:29.974464 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:29.974464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:29.974466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:29.974539 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.974541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.975297 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.975298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.976021 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:29.976023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:29.976661 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:29.976662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:29.976747 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:29.976751 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:29.976809 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:04:29.976848 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260928 09:04:29.977305 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabe40ea32c49918f5fc0d4ebeefe1b" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" is_pre_election: true
I20260928 09:04:29.977485 2419 raft_consensus.cc:2454] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 0.
I20260928 09:04:29.977595 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabe40ea32c49918f5fc0d4ebeefe1b" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be" is_pre_election: true
I20260928 09:04:29.977751 2559 raft_consensus.cc:2454] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 0.
I20260928 09:04:29.977763 2213 leader_election.cc:302] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, 6fc9ed8f250f430da9ae99ed24d14c57; no voters:
I20260928 09:04:29.977895 2634 raft_consensus.cc:2781] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:29.977945 2634 raft_consensus.cc:482] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:29.977970 2634 raft_consensus.cc:3037] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.037022 2634 raft_consensus.cc:504] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.037345 2634 leader_election.cc:288] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:30.037564 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabe40ea32c49918f5fc0d4ebeefe1b" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
I20260928 09:04:30.037612 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acabe40ea32c49918f5fc0d4ebeefe1b" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be"
I20260928 09:04:30.037688 2419 raft_consensus.cc:3037] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.037693 2559 raft_consensus.cc:3037] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.038745 2419 raft_consensus.cc:2454] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 1.
I20260928 09:04:30.038757 2559 raft_consensus.cc:2454] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 1.
I20260928 09:04:30.038960 2213 leader_election.cc:302] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, 6fc9ed8f250f430da9ae99ed24d14c57; no voters:
I20260928 09:04:30.039117 2634 raft_consensus.cc:2781] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:30.039310 2634 raft_consensus.cc:686] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Running, Role: LEADER
I20260928 09:04:30.039443 2634 consensus_queue.cc:237] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.040498 2122 catalog_manager.cc:5495] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "47e97f22e1cd407b9c5b851438b60aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } health_report { overall_health: UNKNOWN } } }
Sep 28 09:04:30 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:30.129715 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.122508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.62:46151)
0928 09:04:30.122580 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.122582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.122697 (+ 115us) server_negotiation.cc:407] Connection header received
0928 09:04:30.122789 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.122790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.122801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.122880 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:30.123397 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.123960 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.124666 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.124857 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.124945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.127147 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:30.127148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:30.127148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:30.127156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:30.128460 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:30.128864 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:30.128865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:30.128866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:30.128924 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:30.129118 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:30.129119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:30.129120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:30.129233 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:30.129311 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.129487 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.129613 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":45}
I20260928 09:04:30.129715 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.122399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46151 (local address 127.0.0.1:38082)
0928 09:04:30.122612 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:30.122625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:30.122707 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:30.122884 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:30.122885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:30.123232 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:30.123279 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.123281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.124046 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.124047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.124580 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.124582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.124945 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.124945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.125016 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.125584 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:30.125591 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:30.127052 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:30.128569 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:30.128578 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:30.128581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:30.128775 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:30.128996 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:30.128997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:30.128997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:30.129056 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:30.129302 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:30.129310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:30.129357 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 09:04:30.129441 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:04:30.129583 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 09:04:30.131798 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38082:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 09:04:30.132066 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:30.133594 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 1a97cbb597bb4349ad0a0c227336bfd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c0034c5dc424bd19ae40a4266b0b5fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:30.133618 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 1a97cbb597bb4349ad0a0c227336bfd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c0034c5dc424bd19ae40a4266b0b5fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:30.133687 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 1a97cbb597bb4349ad0a0c227336bfd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c0034c5dc424bd19ae40a4266b0b5fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:30.133787 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a97cbb597bb4349ad0a0c227336bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:30.133793 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a97cbb597bb4349ad0a0c227336bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:30.133814 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a97cbb597bb4349ad0a0c227336bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:30.134959 2630 tablet_bootstrap.cc:492] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: Bootstrap starting.
I20260928 09:04:30.135280 2633 tablet_bootstrap.cc:492] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: Bootstrap starting.
I20260928 09:04:30.135401 2630 tablet_bootstrap.cc:654] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:30.135411 2628 tablet_bootstrap.cc:492] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: Bootstrap starting.
I20260928 09:04:30.135646 2633 tablet_bootstrap.cc:654] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:30.135975 2628 tablet_bootstrap.cc:654] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:30.136349 2633 tablet_bootstrap.cc:492] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: No bootstrap required, opened a new log
I20260928 09:04:30.136400 2633 ts_tablet_manager.cc:1397] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:30.136535 2633 raft_consensus.cc:348] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.136610 2633 raft_consensus.cc:374] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:30.136632 2633 raft_consensus.cc:729] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc9ed8f250f430da9ae99ed24d14c57, State: Initialized, Role: FOLLOWER
I20260928 09:04:30.136651 2630 tablet_bootstrap.cc:492] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: No bootstrap required, opened a new log
I20260928 09:04:30.136688 2630 ts_tablet_manager.cc:1397] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 09:04:30.136674 2633 consensus_queue.cc:260] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.136806 2633 ts_tablet_manager.cc:1428] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:30.136869 2636 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.136960 2636 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.137027 2630 raft_consensus.cc:348] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.137104 2630 raft_consensus.cc:374] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:30.137130 2630 raft_consensus.cc:729] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Initialized, Role: FOLLOWER
I20260928 09:04:30.137183 2630 consensus_queue.cc:260] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.137282 2630 ts_tablet_manager.cc:1428] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:30.137408 2634 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.137481 2634 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.137750 2628 tablet_bootstrap.cc:492] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: No bootstrap required, opened a new log
I20260928 09:04:30.137800 2628 ts_tablet_manager.cc:1397] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:30.137953 2628 raft_consensus.cc:348] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.138055 2628 raft_consensus.cc:374] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:30.138080 2628 raft_consensus.cc:729] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da659d13c3a4452e9efcf1c98b3f95be, State: Initialized, Role: FOLLOWER
I20260928 09:04:30.138140 2628 consensus_queue.cc:260] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.138226 2628 ts_tablet_manager.cc:1428] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:30.138299 2631 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.138397 2631 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.176028 2636 raft_consensus.cc:482] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:30.176131 2636 raft_consensus.cc:504] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.176590 2636 leader_election.cc:288] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:30.180956 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.177016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33699 (local address 127.0.98.2:41065)
0928 09:04:30.177105 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:30.177116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:30.177174 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:30.177334 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:30.177335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:30.177337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:30.177441 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.177442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.178429 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.178430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.179807 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.179811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.180632 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.180634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.180693 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.180697 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:30.180797 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 09:04:30.180846 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":42,"thread_start_us":51,"threads_started":1}
I20260928 09:04:30.181032 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.177100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41065 (local address 127.0.98.3:33699)
0928 09:04:30.177147 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.177149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.177161 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:30.177236 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.177237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.177246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.177305 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:30.177559 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.178313 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.179964 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.180518 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.180600 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.180612 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.180883 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.180925 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 09:04:30.181197 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.176884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47829 (local address 127.0.98.1:38391)
0928 09:04:30.176963 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.176965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.177177 (+ 212us) server_negotiation.cc:407] Connection header received
0928 09:04:30.177229 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.177230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.177251 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.177313 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:30.177559 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.178313 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.180096 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.180670 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.180730 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.180763 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.181075 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.181125 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:04:30.181143 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.176800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38391 (local address 127.0.98.2:47829)
0928 09:04:30.176974 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:30.176987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:30.177083 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:30.177334 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:30.177335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:30.177340 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:30.177440 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.177442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.178429 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.178430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.180017 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.180019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.180768 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.180769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.180928 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.180932 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:30.180996 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 09:04:30.181034 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":93,"thread_start_us":89,"threads_started":1}
I20260928 09:04:30.181701 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a97cbb597bb4349ad0a0c227336bfd8" candidate_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be" is_pre_election: true
I20260928 09:04:30.181723 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a97cbb597bb4349ad0a0c227336bfd8" candidate_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e97f22e1cd407b9c5b851438b60aa9" is_pre_election: true
I20260928 09:04:30.181828 2559 raft_consensus.cc:2454] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc9ed8f250f430da9ae99ed24d14c57 in term 0.
I20260928 09:04:30.181950 2279 raft_consensus.cc:2454] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc9ed8f250f430da9ae99ed24d14c57 in term 0.
I20260928 09:04:30.182001 2351 leader_election.cc:302] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fc9ed8f250f430da9ae99ed24d14c57, da659d13c3a4452e9efcf1c98b3f95be; no voters:
I20260928 09:04:30.182132 2636 raft_consensus.cc:2781] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:30.182173 2636 raft_consensus.cc:482] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:30.182194 2636 raft_consensus.cc:3037] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.194175 2636 raft_consensus.cc:504] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.194353 2636 leader_election.cc:288] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [CANDIDATE]: Term 1 election: Requested vote from peers 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:30.194571 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a97cbb597bb4349ad0a0c227336bfd8" candidate_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e97f22e1cd407b9c5b851438b60aa9"
I20260928 09:04:30.194684 2279 raft_consensus.cc:3037] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.194628 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a97cbb597bb4349ad0a0c227336bfd8" candidate_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be"
I20260928 09:04:30.194729 2559 raft_consensus.cc:3037] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.195401 2279 raft_consensus.cc:2454] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc9ed8f250f430da9ae99ed24d14c57 in term 1.
I20260928 09:04:30.195688 2351 leader_election.cc:302] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, 6fc9ed8f250f430da9ae99ed24d14c57; no voters:
I20260928 09:04:30.195688 2559 raft_consensus.cc:2454] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc9ed8f250f430da9ae99ed24d14c57 in term 1.
I20260928 09:04:30.195815 2636 raft_consensus.cc:2781] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:30.195958 2636 raft_consensus.cc:686] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc9ed8f250f430da9ae99ed24d14c57, State: Running, Role: LEADER
I20260928 09:04:30.196071 2636 consensus_queue.cc:237] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.196605 2652 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: TxnStatusTablet state changed. Reason: New leader 6fc9ed8f250f430da9ae99ed24d14c57. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.196766 2652 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:30.196874 2122 catalog_manager.cc:5495] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:30.197050 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 09:04:30.197150 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 09:04:30.213914 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.211197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.0.1:51276)
0928 09:04:30.211272 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:30.211285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:30.211360 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:30.211525 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:30.211526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:30.211527 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:30.211581 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.211583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.212288 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.212289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.212893 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.212895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.213313 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.213314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.213389 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.213404 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.213719 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.213729 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 09:04:30.213781 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:04:30.213833 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 09:04:30.213893 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.211293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.0.98.2:43903)
0928 09:04:30.211369 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.211372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.211404 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:04:30.211440 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.211441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.211459 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.211516 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:30.211660 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.212205 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.213013 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.213224 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.213311 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.213506 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.213651 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.213747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.213787 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.213811 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 09:04:30.215750 2559 raft_consensus.cc:1261] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Refusing update from remote peer 6fc9ed8f250f430da9ae99ed24d14c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:30.215807 2279 raft_consensus.cc:1261] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Refusing update from remote peer 6fc9ed8f250f430da9ae99ed24d14c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:30.216118 2636 consensus_queue.cc:1035] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:30.216243 2652 consensus_queue.cc:1035] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:30.217480 2634 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: TxnStatusTablet state changed. Reason: New leader 6fc9ed8f250f430da9ae99ed24d14c57. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.217473 2631 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: TxnStatusTablet state changed. Reason: New leader 6fc9ed8f250f430da9ae99ed24d14c57. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.217594 2634 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.217597 2631 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.218645 2652 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.218734 2652 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:30.218781 2631 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.218865 2631 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P da659d13c3a4452e9efcf1c98b3f95be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.218899 2652 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.218974 2652 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 6fc9ed8f250f430da9ae99ed24d14c57: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:30.219085 2657 mvcc.cc:205] Tried to move back new op lower bound from 7334241362800955392 to 7334241362723762176. Current Snapshot: MvccSnapshot[applied={T|T < 7334241362800955392}]
I20260928 09:04:30.219422 2634 tablet_replica.cc:439] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } } }
I20260928 09:04:30.219516 2634 tablet_replica.cc:442] T 1a97cbb597bb4349ad0a0c227336bfd8 P 47e97f22e1cd407b9c5b851438b60aa9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:30.220007 2658 mvcc.cc:205] Tried to move back new op lower bound from 7334241362800955392 to 7334241362723762176. Current Snapshot: MvccSnapshot[applied={T|T < 7334241362800955392 or (T in {7334241362800955392})}]
I20260928 09:04:30.224710 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.222038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.0.98.1:38391)
0928 09:04:30.222080 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.222082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.222089 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:30.222161 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.222162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.222179 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.222245 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:30.222418 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.223003 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.223873 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.224118 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.224205 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.224386 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.224492 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.224585 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.224604 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.224628 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:30.230585 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.227847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45697 (local address 127.0.98.2:43903)
0928 09:04:30.227894 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.227896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.228003 (+ 107us) server_negotiation.cc:407] Connection header received
0928 09:04:30.228084 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.228085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.228094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.228142 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:30.228311 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.228765 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.229939 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.230115 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.230166 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.230366 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.230396 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.230451 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.230513 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.230530 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 09:04:30.230656 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.227753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.98.1:45697)
0928 09:04:30.227921 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:30.227934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:30.228014 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:30.228162 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:30.228163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:30.228164 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:30.228245 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.228247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.228847 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.228848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.229845 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.229847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.230187 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.230188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.230256 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.230273 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.230454 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.230456 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:04:30.230510 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:04:30.230551 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260928 09:04:30.233053 2559 raft_consensus.cc:1261] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Refusing update from remote peer 47e97f22e1cd407b9c5b851438b60aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:30.233119 2419 raft_consensus.cc:1261] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Refusing update from remote peer 47e97f22e1cd407b9c5b851438b60aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:30.233301 2639 consensus_queue.cc:1035] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:30.233422 2634 consensus_queue.cc:1035] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:30.234524 2659 mvcc.cc:205] Tried to move back new op lower bound from 7334241362872733696 to 7334241362082230272. Current Snapshot: MvccSnapshot[applied={T|T < 7334241362872733696}]
I20260928 09:04:30.242074 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.239259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57519 (local address 127.0.98.1:38391)
0928 09:04:30.239297 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:30.239300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:30.239430 (+ 130us) server_negotiation.cc:407] Connection header received
0928 09:04:30.239506 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:30.239506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:30.239519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:30.239573 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:30.239720 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.240210 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.241264 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.241523 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.241635 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.241818 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.241851 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.241897 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:30.241970 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 09:04:30.241994 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:04:30.242096 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:30.239202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38391 (local address 127.0.98.2:57519)
0928 09:04:30.239354 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:30.239366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:30.239456 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:30.239576 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:30.239578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:30.239579 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:30.239649 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.239651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.240299 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.240300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.241170 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:30.241172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:30.241634 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:30.241635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:30.241702 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:30.241722 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:30.241917 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:30.241918 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:30.241958 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 09:04:30.242000 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260928 09:04:30.250998 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:30.251120 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:30.252399 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Sending DeleteTablet for 3 replicas of tablet acabe40ea32c49918f5fc0d4ebeefe1b
I20260928 09:04:30.252712 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet acabe40ea32c49918f5fc0d4ebeefe1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51270
I20260928 09:04:30.252892 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet acabe40ea32c49918f5fc0d4ebeefe1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49734
I20260928 09:04:30.253103 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet acabe40ea32c49918f5fc0d4ebeefe1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56596
I20260928 09:04:30.253264 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:30.253419 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:30.253764 2680 tablet_replica.cc:330] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: stopping tablet replica
I20260928 09:04:30.254092 2679 tablet_replica.cc:330] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: stopping tablet replica
I20260928 09:04:30.254191 2679 raft_consensus.cc:2229] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:30.254240 2679 raft_consensus.cc:2258] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:30.253875 2680 raft_consensus.cc:2229] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:30.254400 2680 raft_consensus.cc:2258] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:30.254558 2681 tablet_replica.cc:330] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: stopping tablet replica
I20260928 09:04:30.254655 2681 raft_consensus.cc:2229] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:30.254704 2681 raft_consensus.cc:2258] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:30.255159 2680 ts_tablet_manager.cc:1905] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:30.255402 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 6735393e3e1a43d3bcec9fecc2e953ce (DEFAULT_TABLE table=test-table [id=04b587df68da41f9957c820c67128dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:30.255523 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6735393e3e1a43d3bcec9fecc2e953ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:30.256829 2633 tablet_bootstrap.cc:492] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Bootstrap starting.
I20260928 09:04:30.257233 2633 tablet_bootstrap.cc:654] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:30.257318 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 6735393e3e1a43d3bcec9fecc2e953ce (DEFAULT_TABLE table=test-table [id=04b587df68da41f9957c820c67128dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:30.257428 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6735393e3e1a43d3bcec9fecc2e953ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:30.257876 2633 tablet_bootstrap.cc:492] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: No bootstrap required, opened a new log
I20260928 09:04:30.257928 2633 ts_tablet_manager.cc:1397] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:30.257941 2679 ts_tablet_manager.cc:1905] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:30.258060 2633 raft_consensus.cc:348] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.258136 2633 raft_consensus.cc:374] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:30.258157 2633 raft_consensus.cc:729] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc9ed8f250f430da9ae99ed24d14c57, State: Initialized, Role: FOLLOWER
I20260928 09:04:30.258200 2633 consensus_queue.cc:260] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.258281 2633 ts_tablet_manager.cc:1428] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:30.258687 2628 tablet_bootstrap.cc:492] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Bootstrap starting.
I20260928 09:04:30.259052 2628 tablet_bootstrap.cc:654] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:30.259407 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 6735393e3e1a43d3bcec9fecc2e953ce (DEFAULT_TABLE table=test-table [id=04b587df68da41f9957c820c67128dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:30.260303 2628 tablet_bootstrap.cc:492] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: No bootstrap required, opened a new log
I20260928 09:04:30.260360 2628 ts_tablet_manager.cc:1397] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:30.260504 2628 raft_consensus.cc:348] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.260578 2628 raft_consensus.cc:374] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:30.260599 2628 raft_consensus.cc:729] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da659d13c3a4452e9efcf1c98b3f95be, State: Initialized, Role: FOLLOWER
I20260928 09:04:30.260644 2628 consensus_queue.cc:260] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.260730 2628 ts_tablet_manager.cc:1428] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:30.261879 2681 ts_tablet_manager.cc:1905] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:30.276316 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6735393e3e1a43d3bcec9fecc2e953ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:30.277709 2630 tablet_bootstrap.cc:492] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Bootstrap starting.
I20260928 09:04:30.278151 2630 tablet_bootstrap.cc:654] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:30.278815 2630 tablet_bootstrap.cc:492] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: No bootstrap required, opened a new log
I20260928 09:04:30.278867 2630 ts_tablet_manager.cc:1397] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:30.279023 2630 raft_consensus.cc:348] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.279106 2630 raft_consensus.cc:374] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:30.279125 2630 raft_consensus.cc:729] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Initialized, Role: FOLLOWER
I20260928 09:04:30.279167 2630 consensus_queue.cc:260] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.279249 2630 ts_tablet_manager.cc:1428] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:30.279289 2680 ts_tablet_manager.cc:1918] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:30.279366 2680 log.cc:1196] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/acabe40ea32c49918f5fc0d4ebeefe1b
I20260928 09:04:30.279629 2680 ts_tablet_manager.cc:1939] T acabe40ea32c49918f5fc0d4ebeefe1b P 47e97f22e1cd407b9c5b851438b60aa9: Deleting consensus metadata
I20260928 09:04:30.289144 2679 ts_tablet_manager.cc:1918] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:30.289255 2679 log.cc:1196] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/acabe40ea32c49918f5fc0d4ebeefe1b
I20260928 09:04:30.289534 2679 ts_tablet_manager.cc:1939] T acabe40ea32c49918f5fc0d4ebeefe1b P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting consensus metadata
I20260928 09:04:30.307410 2109 catalog_manager.cc:4841] TS 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903): tablet acabe40ea32c49918f5fc0d4ebeefe1b (table test-table [id=b023392ee56c4b768f1b28eaae134c4d]) successfully deleted
I20260928 09:04:30.308574 2672 raft_consensus.cc:482] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:30.308669 2672 raft_consensus.cc:504] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.308893 2672 leader_election.cc:288] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:30.309116 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6735393e3e1a43d3bcec9fecc2e953ce" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be" is_pre_election: true
I20260928 09:04:30.309226 2559 raft_consensus.cc:2454] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 0.
I20260928 09:04:30.309374 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6735393e3e1a43d3bcec9fecc2e953ce" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" is_pre_election: true
I20260928 09:04:30.309455 2419 raft_consensus.cc:2454] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 0.
I20260928 09:04:30.309656 2211 leader_election.cc:302] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, da659d13c3a4452e9efcf1c98b3f95be; no voters:
I20260928 09:04:30.309795 2672 raft_consensus.cc:2781] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:30.309832 2672 raft_consensus.cc:482] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:30.309851 2672 raft_consensus.cc:3037] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.310755 2672 raft_consensus.cc:504] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.310916 2672 leader_election.cc:288] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:30.311064 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6735393e3e1a43d3bcec9fecc2e953ce" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
I20260928 09:04:30.311138 2419 raft_consensus.cc:3037] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.311913 2419 raft_consensus.cc:2454] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 1.
I20260928 09:04:30.312093 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6735393e3e1a43d3bcec9fecc2e953ce" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be"
I20260928 09:04:30.312158 2559 raft_consensus.cc:3037] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:30.312140 2213 leader_election.cc:302] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, 6fc9ed8f250f430da9ae99ed24d14c57; no voters:
I20260928 09:04:30.312222 2672 raft_consensus.cc:2781] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:30.312259 2672 raft_consensus.cc:686] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Running, Role: LEADER
I20260928 09:04:30.312310 2672 consensus_queue.cc:237] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:30.312867 2559 raft_consensus.cc:2454] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 1.
I20260928 09:04:30.312882 2122 catalog_manager.cc:5495] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "47e97f22e1cd407b9c5b851438b60aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:30.321000 2681 ts_tablet_manager.cc:1918] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:30.321139 2681 log.cc:1196] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/acabe40ea32c49918f5fc0d4ebeefe1b
I20260928 09:04:30.321439 2681 ts_tablet_manager.cc:1939] T acabe40ea32c49918f5fc0d4ebeefe1b P da659d13c3a4452e9efcf1c98b3f95be: Deleting consensus metadata
I20260928 09:04:30.322921 2107 catalog_manager.cc:4841] TS da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699): tablet acabe40ea32c49918f5fc0d4ebeefe1b (table test-table [id=b023392ee56c4b768f1b28eaae134c4d]) successfully deleted
I20260928 09:04:30.323016 2107 catalog_manager.cc:4841] TS 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391): tablet acabe40ea32c49918f5fc0d4ebeefe1b (table test-table [id=b023392ee56c4b768f1b28eaae134c4d]) successfully deleted
I20260928 09:04:30.323671 2559 raft_consensus.cc:1261] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Refusing update from remote peer 47e97f22e1cd407b9c5b851438b60aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:30.323712 2419 raft_consensus.cc:1261] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Refusing update from remote peer 47e97f22e1cd407b9c5b851438b60aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:30.323880 2639 consensus_queue.cc:1035] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:30.323983 2672 consensus_queue.cc:1035] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 09:04:30.329650 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 09:04:31.851375 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:31.851483 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:31.852277 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Sending DeleteTablet for 3 replicas of tablet 6735393e3e1a43d3bcec9fecc2e953ce
I20260928 09:04:31.852588 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 6735393e3e1a43d3bcec9fecc2e953ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49734
I20260928 09:04:31.852640 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 6735393e3e1a43d3bcec9fecc2e953ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51270
I20260928 09:04:31.852793 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 6735393e3e1a43d3bcec9fecc2e953ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56596
I20260928 09:04:31.852988 2711 tablet_replica.cc:330] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: stopping tablet replica
I20260928 09:04:31.853077 2711 raft_consensus.cc:2229] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:31.853127 2711 raft_consensus.cc:2258] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:31.853149 2709 tablet_replica.cc:330] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: stopping tablet replica
I20260928 09:04:31.853220 2709 raft_consensus.cc:2229] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:31.853291 2709 raft_consensus.cc:2258] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:31.853405 2710 tablet_replica.cc:330] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: stopping tablet replica
I20260928 09:04:31.853402 2711 ts_tablet_manager.cc:1905] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:31.853468 2710 raft_consensus.cc:2229] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:31.853574 2710 raft_consensus.cc:2258] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:31.853850 2710 ts_tablet_manager.cc:1905] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:31.855542 2711 ts_tablet_manager.cc:1918] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 09:04:31.855641 2711 log.cc:1196] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/6735393e3e1a43d3bcec9fecc2e953ce
I20260928 09:04:31.855922 2711 ts_tablet_manager.cc:1939] T 6735393e3e1a43d3bcec9fecc2e953ce P da659d13c3a4452e9efcf1c98b3f95be: Deleting consensus metadata
I20260928 09:04:31.856112 2710 ts_tablet_manager.cc:1918] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 09:04:31.856174 2710 log.cc:1196] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/6735393e3e1a43d3bcec9fecc2e953ce
I20260928 09:04:31.856331 2107 catalog_manager.cc:4841] TS da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699): tablet 6735393e3e1a43d3bcec9fecc2e953ce (table test-table [id=04b587df68da41f9957c820c67128dec]) successfully deleted
I20260928 09:04:31.856439 2710 ts_tablet_manager.cc:1939] T 6735393e3e1a43d3bcec9fecc2e953ce P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting consensus metadata
I20260928 09:04:31.856671 2709 ts_tablet_manager.cc:1905] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:31.857052 2109 catalog_manager.cc:4841] TS 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903): tablet 6735393e3e1a43d3bcec9fecc2e953ce (table test-table [id=04b587df68da41f9957c820c67128dec]) successfully deleted
I20260928 09:04:31.858170 2709 ts_tablet_manager.cc:1918] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 09:04:31.858220 2709 log.cc:1196] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/6735393e3e1a43d3bcec9fecc2e953ce
I20260928 09:04:31.858454 2709 ts_tablet_manager.cc:1939] T 6735393e3e1a43d3bcec9fecc2e953ce P 47e97f22e1cd407b9c5b851438b60aa9: Deleting consensus metadata
I20260928 09:04:31.858830 2107 catalog_manager.cc:4841] TS 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391): tablet 6735393e3e1a43d3bcec9fecc2e953ce (table test-table [id=04b587df68da41f9957c820c67128dec]) successfully deleted
Sep 28 09:04:31 dist-test-slave-smns krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586271, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 09:04:31 dist-test-slave-smns krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586271, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:31.878396 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.871141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.0.98.62:46151)
0928 09:04:31.871301 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.871304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.871344 (+ 40us) server_negotiation.cc:407] Connection header received
0928 09:04:31.871379 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.871380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.871395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.871457 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:31.872026 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.872621 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.873441 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.873750 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.873814 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.876119 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:31.876121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:31.876121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:31.876128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:31.877407 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:31.877779 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:31.877780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:31.877780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:31.877834 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:31.878002 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:31.878003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:31.878003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:31.878099 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:31.878168 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.878263 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.878325 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 09:04:31.879864 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:31.880038 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:31.881592 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 33d307b6d86b49ea8c764dac6f66b2cb (DEFAULT_TABLE table=test-table [id=b7d99b05617044bd82ae72f129604630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:31.881633 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 33d307b6d86b49ea8c764dac6f66b2cb (DEFAULT_TABLE table=test-table [id=b7d99b05617044bd82ae72f129604630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:31.881736 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d307b6d86b49ea8c764dac6f66b2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:31.881742 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 33d307b6d86b49ea8c764dac6f66b2cb (DEFAULT_TABLE table=test-table [id=b7d99b05617044bd82ae72f129604630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:31.881779 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d307b6d86b49ea8c764dac6f66b2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:31.881841 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d307b6d86b49ea8c764dac6f66b2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:31.882934 2720 tablet_bootstrap.cc:492] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Bootstrap starting.
I20260928 09:04:31.883266 2721 tablet_bootstrap.cc:492] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Bootstrap starting.
I20260928 09:04:31.883299 2720 tablet_bootstrap.cc:654] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:31.883339 2722 tablet_bootstrap.cc:492] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Bootstrap starting.
I20260928 09:04:31.883667 2721 tablet_bootstrap.cc:654] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:31.883718 2722 tablet_bootstrap.cc:654] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:31.883991 2720 tablet_bootstrap.cc:492] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: No bootstrap required, opened a new log
I20260928 09:04:31.884049 2720 ts_tablet_manager.cc:1397] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:31.884243 2720 raft_consensus.cc:348] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.884325 2721 tablet_bootstrap.cc:492] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: No bootstrap required, opened a new log
I20260928 09:04:31.884339 2720 raft_consensus.cc:374] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:31.884362 2720 raft_consensus.cc:729] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da659d13c3a4452e9efcf1c98b3f95be, State: Initialized, Role: FOLLOWER
I20260928 09:04:31.884372 2721 ts_tablet_manager.cc:1397] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:31.884388 2722 tablet_bootstrap.cc:492] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: No bootstrap required, opened a new log
I20260928 09:04:31.884413 2720 consensus_queue.cc:260] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.884459 2722 ts_tablet_manager.cc:1397] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:31.884485 2721 raft_consensus.cc:348] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.884572 2721 raft_consensus.cc:374] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:31.884591 2721 raft_consensus.cc:729] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Initialized, Role: FOLLOWER
I20260928 09:04:31.884601 2720 ts_tablet_manager.cc:1428] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:31.884582 2722 raft_consensus.cc:348] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.884650 2722 raft_consensus.cc:374] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:31.884634 2721 consensus_queue.cc:260] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.884670 2722 raft_consensus.cc:729] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc9ed8f250f430da9ae99ed24d14c57, State: Initialized, Role: FOLLOWER
I20260928 09:04:31.884714 2722 consensus_queue.cc:260] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.884727 2721 ts_tablet_manager.cc:1428] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:31.885087 2722 ts_tablet_manager.cc:1428] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 09:04:31.887727 2726 raft_consensus.cc:482] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:31.887791 2726 raft_consensus.cc:504] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.888103 2726 leader_election.cc:288] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391), 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903)
I20260928 09:04:31.891521 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.888413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50627 (local address 127.0.98.2:43903)
0928 09:04:31.888560 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.888562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.888570 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:31.888594 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.888595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.888602 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.888655 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:31.888826 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.889487 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.890758 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.891145 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.891239 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.891251 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.891410 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.891454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 09:04:31.891527 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.888238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.98.3:50627)
0928 09:04:31.888387 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:31.888407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:31.888471 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:31.888681 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:31.888682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:31.888685 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:31.888753 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.888755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.889595 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.889596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.890691 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.890694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.891241 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.891242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.891330 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.891334 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:31.891404 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 09:04:31.891446 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260928 09:04:31.891866 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.888268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38053 (local address 127.0.98.1:38391)
0928 09:04:31.888563 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.888565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.888572 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:31.888600 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.888600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.888608 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.888686 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:31.889239 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.889897 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.891012 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.891472 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.891561 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.891573 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.891739 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.891784 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":69,"threads_started":1}
I20260928 09:04:31.891862 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.888230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38391 (local address 127.0.98.3:38053)
0928 09:04:31.888377 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:31.888388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:31.888451 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:31.888702 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:31.888703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:31.888705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:31.889147 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.889150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.889976 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.889976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.890919 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.890921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.891565 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.891566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.891637 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.891641 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:31.891725 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:04:31.891764 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":104,"threads_started":1}
I20260928 09:04:31.891981 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d307b6d86b49ea8c764dac6f66b2cb" candidate_uuid: "da659d13c3a4452e9efcf1c98b3f95be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" is_pre_election: true
I20260928 09:04:31.892102 2419 raft_consensus.cc:2454] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da659d13c3a4452e9efcf1c98b3f95be in term 0.
I20260928 09:04:31.892119 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d307b6d86b49ea8c764dac6f66b2cb" candidate_uuid: "da659d13c3a4452e9efcf1c98b3f95be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e97f22e1cd407b9c5b851438b60aa9" is_pre_election: true
I20260928 09:04:31.892198 2279 raft_consensus.cc:2454] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da659d13c3a4452e9efcf1c98b3f95be in term 0.
I20260928 09:04:31.892284 2493 leader_election.cc:302] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fc9ed8f250f430da9ae99ed24d14c57, da659d13c3a4452e9efcf1c98b3f95be; no voters:
I20260928 09:04:31.892405 2726 raft_consensus.cc:2781] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:31.892467 2726 raft_consensus.cc:482] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:31.892483 2726 raft_consensus.cc:3037] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:31.893189 2726 raft_consensus.cc:504] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.893343 2726 leader_election.cc:288] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [CANDIDATE]: Term 1 election: Requested vote from peers 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391), 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903)
I20260928 09:04:31.893468 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d307b6d86b49ea8c764dac6f66b2cb" candidate_uuid: "da659d13c3a4452e9efcf1c98b3f95be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e97f22e1cd407b9c5b851438b60aa9"
I20260928 09:04:31.893595 2279 raft_consensus.cc:3037] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:31.893592 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d307b6d86b49ea8c764dac6f66b2cb" candidate_uuid: "da659d13c3a4452e9efcf1c98b3f95be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
I20260928 09:04:31.893672 2419 raft_consensus.cc:3037] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:31.894342 2279 raft_consensus.cc:2454] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da659d13c3a4452e9efcf1c98b3f95be in term 1.
I20260928 09:04:31.894474 2419 raft_consensus.cc:2454] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da659d13c3a4452e9efcf1c98b3f95be in term 1.
I20260928 09:04:31.894516 2491 leader_election.cc:302] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, da659d13c3a4452e9efcf1c98b3f95be; no voters:
I20260928 09:04:31.894603 2726 raft_consensus.cc:2781] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:31.894747 2726 raft_consensus.cc:686] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 1 LEADER]: Becoming Leader. State: Replica: da659d13c3a4452e9efcf1c98b3f95be, State: Running, Role: LEADER
I20260928 09:04:31.894801 2726 consensus_queue.cc:237] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.895484 2121 catalog_manager.cc:5495] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be reported cstate change: term changed from 0 to 1, leader changed from <none> to da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3). New cstate: current_term: 1 leader_uuid: "da659d13c3a4452e9efcf1c98b3f95be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } health_report { overall_health: HEALTHY } } }
I20260928 09:04:31.904599 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.902004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.0.98.3:33699)
0928 09:04:31.902151 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.902153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.902160 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:31.902199 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.902200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.902217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.902307 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:31.902433 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.902933 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.903740 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.903971 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.904047 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.904228 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:31.904343 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:31.904424 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.904485 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.904506 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260928 09:04:31.909327 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.907219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37151 (local address 127.0.98.2:43903)
0928 09:04:31.907316 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.907318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.907355 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:04:31.907446 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.907446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.907479 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.907529 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:31.907639 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.908082 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.908803 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.908975 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.909023 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.909164 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:31.909187 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:31.909220 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.909261 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.909276 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 09:04:31.909354 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.907146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43903 (local address 127.0.98.3:37151)
0928 09:04:31.907308 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:31.907317 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:31.907367 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:31.907543 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:31.907544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:31.907545 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:31.907592 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.907593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.908145 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.908145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.908735 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.908737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.909039 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.909040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.909085 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.909098 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:31.909228 (+ 130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:31.909229 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:31.909258 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 09:04:31.909289 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260928 09:04:31.911362 2419 raft_consensus.cc:1261] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Refusing update from remote peer da659d13c3a4452e9efcf1c98b3f95be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:31.911392 2279 raft_consensus.cc:1261] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Refusing update from remote peer da659d13c3a4452e9efcf1c98b3f95be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:31.911636 2726 consensus_queue.cc:1035] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:31.911764 2731 consensus_queue.cc:1035] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:31.918735 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.916104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33699 (local address 127.0.98.2:53465)
0928 09:04:31.916230 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:31.916241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:31.916293 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:31.916417 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:31.916417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:31.916419 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:31.916456 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.916458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.917178 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.917179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.917823 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:31.917826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.918345 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.918346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:31.918407 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.918412 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:31.918611 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:31.918613 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:04:31.918641 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 09:04:31.918679 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":86,"threads_started":1}
I20260928 09:04:31.918756 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.916138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53465 (local address 127.0.98.3:33699)
0928 09:04:31.916175 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.916177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.916281 (+ 104us) server_negotiation.cc:407] Connection header received
0928 09:04:31.916341 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.916342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.916352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.916400 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:31.916526 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.917084 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.917964 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.918244 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.918345 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.918542 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:31.918568 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:31.918609 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.918648 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.918665 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:04:31.925231 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:31.925341 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:31.926040 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Sending DeleteTablet for 3 replicas of tablet 33d307b6d86b49ea8c764dac6f66b2cb
I20260928 09:04:31.926306 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d307b6d86b49ea8c764dac6f66b2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56596
I20260928 09:04:31.926342 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d307b6d86b49ea8c764dac6f66b2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51270
I20260928 09:04:31.926419 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d307b6d86b49ea8c764dac6f66b2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49734
I20260928 09:04:31.926479 2710 tablet_replica.cc:330] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: stopping tablet replica
I20260928 09:04:31.926504 2711 tablet_replica.cc:330] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: stopping tablet replica
I20260928 09:04:31.926546 2711 raft_consensus.cc:2229] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:31.926544 2710 raft_consensus.cc:2229] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:31.926585 2710 raft_consensus.cc:2258] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:31.926622 2711 raft_consensus.cc:2258] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:31.926693 2709 tablet_replica.cc:330] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: stopping tablet replica
I20260928 09:04:31.926765 2709 raft_consensus.cc:2229] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:31.926803 2709 raft_consensus.cc:2258] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:31.926848 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 09:04:31.926949 2711 ts_tablet_manager.cc:1905] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 09:04:31.927022 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:31.928256 2259 tablet_service.cc:1467] Processing CreateTablet for tablet c874c44903ab4a5e9b0a65bf691398f3 (DEFAULT_TABLE table=test-table [id=656ca19b7965474093a0878265cfb629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:31.928378 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874c44903ab4a5e9b0a65bf691398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:31.929001 2710 ts_tablet_manager.cc:1905] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:31.929196 2711 ts_tablet_manager.cc:1918] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:31.929255 2711 log.cc:1196] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/33d307b6d86b49ea8c764dac6f66b2cb
I20260928 09:04:31.929481 2711 ts_tablet_manager.cc:1939] T 33d307b6d86b49ea8c764dac6f66b2cb P da659d13c3a4452e9efcf1c98b3f95be: Deleting consensus metadata
I20260928 09:04:31.929919 2721 tablet_bootstrap.cc:492] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9: Bootstrap starting.
I20260928 09:04:31.929955 2107 catalog_manager.cc:4841] TS da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699): tablet 33d307b6d86b49ea8c764dac6f66b2cb (table test-table [id=b7d99b05617044bd82ae72f129604630]) successfully deleted
I20260928 09:04:31.930272 2721 tablet_bootstrap.cc:654] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:31.930614 2539 tablet_service.cc:1467] Processing CreateTablet for tablet c874c44903ab4a5e9b0a65bf691398f3 (DEFAULT_TABLE table=test-table [id=656ca19b7965474093a0878265cfb629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:31.930711 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874c44903ab4a5e9b0a65bf691398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:31.930752 2709 ts_tablet_manager.cc:1905] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:31.931936 2720 tablet_bootstrap.cc:492] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Bootstrap starting.
I20260928 09:04:31.931988 2710 ts_tablet_manager.cc:1918] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:31.932034 2710 log.cc:1196] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/33d307b6d86b49ea8c764dac6f66b2cb
I20260928 09:04:31.932142 2709 ts_tablet_manager.cc:1918] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:31.932193 2721 tablet_bootstrap.cc:492] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9: No bootstrap required, opened a new log
I20260928 09:04:31.932242 2721 ts_tablet_manager.cc:1397] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:31.932271 2710 ts_tablet_manager.cc:1939] T 33d307b6d86b49ea8c764dac6f66b2cb P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting consensus metadata
I20260928 09:04:31.932337 2720 tablet_bootstrap.cc:654] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:31.932345 2721 raft_consensus.cc:348] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.932410 2721 raft_consensus.cc:374] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:31.932432 2721 raft_consensus.cc:729] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Initialized, Role: FOLLOWER
I20260928 09:04:31.932475 2721 consensus_queue.cc:260] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.932582 2721 ts_tablet_manager.cc:1428] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:31.932634 2109 catalog_manager.cc:4841] TS 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903): tablet 33d307b6d86b49ea8c764dac6f66b2cb (table test-table [id=b7d99b05617044bd82ae72f129604630]) successfully deleted
I20260928 09:04:31.932673 2399 tablet_service.cc:1467] Processing CreateTablet for tablet c874c44903ab4a5e9b0a65bf691398f3 (DEFAULT_TABLE table=test-table [id=656ca19b7965474093a0878265cfb629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:31.932765 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874c44903ab4a5e9b0a65bf691398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:31.932193 2709 log.cc:1196] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/33d307b6d86b49ea8c764dac6f66b2cb
I20260928 09:04:31.933185 2709 ts_tablet_manager.cc:1939] T 33d307b6d86b49ea8c764dac6f66b2cb P 47e97f22e1cd407b9c5b851438b60aa9: Deleting consensus metadata
I20260928 09:04:31.933481 2720 tablet_bootstrap.cc:492] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: No bootstrap required, opened a new log
I20260928 09:04:31.933525 2720 ts_tablet_manager.cc:1397] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:31.933573 2107 catalog_manager.cc:4841] TS 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391): tablet 33d307b6d86b49ea8c764dac6f66b2cb (table test-table [id=b7d99b05617044bd82ae72f129604630]) successfully deleted
I20260928 09:04:31.933730 2720 raft_consensus.cc:348] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.933827 2720 raft_consensus.cc:374] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:31.933848 2720 raft_consensus.cc:729] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da659d13c3a4452e9efcf1c98b3f95be, State: Initialized, Role: FOLLOWER
I20260928 09:04:31.933892 2720 consensus_queue.cc:260] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.933998 2720 ts_tablet_manager.cc:1428] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:31.934345 2722 tablet_bootstrap.cc:492] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Bootstrap starting.
I20260928 09:04:31.934710 2722 tablet_bootstrap.cc:654] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:31.935232 2722 tablet_bootstrap.cc:492] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: No bootstrap required, opened a new log
I20260928 09:04:31.935274 2722 ts_tablet_manager.cc:1397] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:31.935397 2722 raft_consensus.cc:348] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.935463 2722 raft_consensus.cc:374] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:31.935477 2722 raft_consensus.cc:729] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc9ed8f250f430da9ae99ed24d14c57, State: Initialized, Role: FOLLOWER
I20260928 09:04:31.935530 2722 consensus_queue.cc:260] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.935616 2722 ts_tablet_manager.cc:1428] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:31.962925 2702 raft_consensus.cc:482] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:31.963021 2702 raft_consensus.cc:504] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.963213 2702 leader_election.cc:288] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:31.963402 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874c44903ab4a5e9b0a65bf691398f3" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" is_pre_election: true
I20260928 09:04:31.963519 2419 raft_consensus.cc:2454] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 0.
I20260928 09:04:31.963620 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874c44903ab4a5e9b0a65bf691398f3" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be" is_pre_election: true
I20260928 09:04:31.963689 2213 leader_election.cc:302] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, 6fc9ed8f250f430da9ae99ed24d14c57; no voters:
I20260928 09:04:31.963730 2558 raft_consensus.cc:2454] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 0.
I20260928 09:04:31.963778 2702 raft_consensus.cc:2781] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:31.963807 2702 raft_consensus.cc:482] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:31.963825 2702 raft_consensus.cc:3037] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:31.964581 2702 raft_consensus.cc:504] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.964736 2702 leader_election.cc:288] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903), da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699)
I20260928 09:04:31.964879 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874c44903ab4a5e9b0a65bf691398f3" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da659d13c3a4452e9efcf1c98b3f95be"
I20260928 09:04:31.964977 2558 raft_consensus.cc:3037] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:31.964907 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874c44903ab4a5e9b0a65bf691398f3" candidate_uuid: "47e97f22e1cd407b9c5b851438b60aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc9ed8f250f430da9ae99ed24d14c57"
I20260928 09:04:31.965015 2419 raft_consensus.cc:3037] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:31.965652 2419 raft_consensus.cc:2454] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 1.
I20260928 09:04:31.965809 2558 raft_consensus.cc:2454] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e97f22e1cd407b9c5b851438b60aa9 in term 1.
I20260928 09:04:31.965835 2213 leader_election.cc:302] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e97f22e1cd407b9c5b851438b60aa9, 6fc9ed8f250f430da9ae99ed24d14c57; no voters:
I20260928 09:04:31.965917 2702 raft_consensus.cc:2781] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:31.965961 2702 raft_consensus.cc:686] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 47e97f22e1cd407b9c5b851438b60aa9, State: Running, Role: LEADER
I20260928 09:04:31.966006 2702 consensus_queue.cc:237] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } }
I20260928 09:04:31.966603 2120 catalog_manager.cc:5495] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "47e97f22e1cd407b9c5b851438b60aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e97f22e1cd407b9c5b851438b60aa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:31.982513 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:31.980215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.1:38391)
0928 09:04:31.980254 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:31.980257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:31.980265 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:31.980323 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:31.980324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:31.980336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:31.980389 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:31.980496 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.980967 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.981815 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:31.982017 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:31.982099 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:31.982245 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:31.982280 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:31.982345 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:31.982406 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 09:04:31.982426 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:04:31.984766 2419 raft_consensus.cc:1261] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Refusing update from remote peer 47e97f22e1cd407b9c5b851438b60aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:31.984932 2558 raft_consensus.cc:1261] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Refusing update from remote peer 47e97f22e1cd407b9c5b851438b60aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:31.985088 2702 consensus_queue.cc:1035] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc9ed8f250f430da9ae99ed24d14c57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:31.985194 2672 consensus_queue.cc:1035] T c874c44903ab4a5e9b0a65bf691398f3 P 47e97f22e1cd407b9c5b851438b60aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da659d13c3a4452e9efcf1c98b3f95be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 09:04:31.989739 2693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 09:04:33.508920 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:33.509028 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:33.509879 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f41c4d16ce684330b551d3ca972dc05d: Sending DeleteTablet for 3 replicas of tablet c874c44903ab4a5e9b0a65bf691398f3
I20260928 09:04:33.510170 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet c874c44903ab4a5e9b0a65bf691398f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56596
I20260928 09:04:33.510188 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet c874c44903ab4a5e9b0a65bf691398f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51270
I20260928 09:04:33.510200 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet c874c44903ab4a5e9b0a65bf691398f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49734
I20260928 09:04:33.510308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 2194
I20260928 09:04:33.513646 2772 tablet_replica.cc:330] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: stopping tablet replica
I20260928 09:04:33.513731 2772 raft_consensus.cc:2229] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:33.513787 2772 raft_consensus.cc:2258] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:33.513896 2771 tablet_replica.cc:330] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: stopping tablet replica
I20260928 09:04:33.513969 2771 raft_consensus.cc:2229] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:33.514009 2771 raft_consensus.cc:2258] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:33.514202 2771 ts_tablet_manager.cc:1905] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:33.514206 2772 ts_tablet_manager.cc:1905] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:33.516044 2772 ts_tablet_manager.cc:1918] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 09:04:33.516085 2771 ts_tablet_manager.cc:1918] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 09:04:33.516145 2771 log.cc:1196] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/c874c44903ab4a5e9b0a65bf691398f3
I20260928 09:04:33.516155 2772 log.cc:1196] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/c874c44903ab4a5e9b0a65bf691398f3
I20260928 09:04:33.516372 2772 ts_tablet_manager.cc:1939] T c874c44903ab4a5e9b0a65bf691398f3 P da659d13c3a4452e9efcf1c98b3f95be: Deleting consensus metadata
I20260928 09:04:33.516386 2771 ts_tablet_manager.cc:1939] T c874c44903ab4a5e9b0a65bf691398f3 P 6fc9ed8f250f430da9ae99ed24d14c57: Deleting consensus metadata
I20260928 09:04:33.516778 2109 catalog_manager.cc:4841] TS 6fc9ed8f250f430da9ae99ed24d14c57 (127.0.98.2:43903): tablet c874c44903ab4a5e9b0a65bf691398f3 (table test-table [id=656ca19b7965474093a0878265cfb629]) successfully deleted
I20260928 09:04:33.516795 2107 catalog_manager.cc:4841] TS da659d13c3a4452e9efcf1c98b3f95be (127.0.98.3:33699): tablet c874c44903ab4a5e9b0a65bf691398f3 (table test-table [id=656ca19b7965474093a0878265cfb629]) successfully deleted
W20260928 09:04:33.519091 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38391 (error 108)
I20260928 09:04:33.519261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 2334
I20260928 09:04:33.524731 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:33.519437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38391 (local address 127.0.0.1:57540)
0928 09:04:33.524616 (+ 5179us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:33.524676 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5142,"thread_start_us":66,"threads_started":1}
I20260928 09:04:33.524868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 2474
W20260928 09:04:33.524873 2107 catalog_manager.cc:4584] TS 47e97f22e1cd407b9c5b851438b60aa9 (127.0.98.1:38391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c874c44903ab4a5e9b0a65bf691398f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38391: connect: Connection refused (error 111)
I20260928 09:04:33.531348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 2090
2026-09-28T09:04:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5631 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:33 dist-test-slave-smns krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:33 dist-test-slave-smns krb5kdc[2778](info): set up 2 sockets
Sep 28 09:04:33 dist-test-slave-smns krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260928 09:04:34.577292 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:34 dist-test-slave-smns krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:34.697278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:34.780937 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:34.781183 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:34.781219 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:34.782437 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:34.782513 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:34.782537 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:34.782565 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:34.782593 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:34.784060 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:34.784329 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:34.784550 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:34.789361 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:34.789390 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:34.789361 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:34.789438 2791 server_base.cc:1034] running on GCE node
I20260928 09:04:34.789870 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:34.790169 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:34.791309 2791 hybrid_clock.cc:648] HybridClock initialized: now 1790586274791280 us; error 46 us; skew 500 ppm
Sep 28 09:04:34 dist-test-slave-smns krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:34.804350 2791 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:34.804823 2791 webserver.cc:460] Webserver started at http://127.0.98.62:43899/ using document root <none> and password file <none>
I20260928 09:04:34.805105 2791 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:34.805174 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:34.805318 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:34.806473 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0f7f74261aa04615a9d43902801a95bd"
format_stamp: "Formatted at 2026-09-28 09:04:34 on dist-test-slave-smns"
server_key: "40b3c46bcba043501d6eabc12d35a851"
server_key_iv: "a0e6e3753e4c4fc28e1c5b0b1fd5655b"
server_key_version: "encryptionkey@0"
I20260928 09:04:34.806865 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f7f74261aa04615a9d43902801a95bd"
format_stamp: "Formatted at 2026-09-28 09:04:34 on dist-test-slave-smns"
server_key: "40b3c46bcba043501d6eabc12d35a851"
server_key_iv: "a0e6e3753e4c4fc28e1c5b0b1fd5655b"
server_key_version: "encryptionkey@0"
I20260928 09:04:34.808226 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:04:34.808957 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:34.809139 2791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:34.809240 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0f7f74261aa04615a9d43902801a95bd"
format_stamp: "Formatted at 2026-09-28 09:04:34 on dist-test-slave-smns"
server_key: "40b3c46bcba043501d6eabc12d35a851"
server_key_iv: "a0e6e3753e4c4fc28e1c5b0b1fd5655b"
server_key_version: "encryptionkey@0"
I20260928 09:04:34.809322 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:34.820143 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:34.822047 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:34.822206 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:34.826395 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41199
I20260928 09:04:34.826431 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41199 every 8 connection(s)
I20260928 09:04:34.826946 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:34.827648 2859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:34.829883 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd: Bootstrap starting.
I20260928 09:04:34.830422 2859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:34.830650 2859 log.cc:824] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:34.831249 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd: No bootstrap required, opened a new log
I20260928 09:04:34.832746 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7f74261aa04615a9d43902801a95bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41199 } }
I20260928 09:04:34.832876 2859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:34.832901 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7f74261aa04615a9d43902801a95bd, State: Initialized, Role: FOLLOWER
I20260928 09:04:34.833035 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7f74261aa04615a9d43902801a95bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41199 } }
I20260928 09:04:34.833118 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:34.833141 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:34.833168 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:34.834069 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7f74261aa04615a9d43902801a95bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41199 } }
I20260928 09:04:34.834152 2859 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f7f74261aa04615a9d43902801a95bd; no voters:
I20260928 09:04:34.834280 2859 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:34.834350 2862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:34.834517 2862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [term 1 LEADER]: Becoming Leader. State: Replica: 0f7f74261aa04615a9d43902801a95bd, State: Running, Role: LEADER
I20260928 09:04:34.834656 2862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7f74261aa04615a9d43902801a95bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41199 } }
I20260928 09:04:34.835073 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f7f74261aa04615a9d43902801a95bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7f74261aa04615a9d43902801a95bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41199 } } }
I20260928 09:04:34.835158 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:34.835289 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7f74261aa04615a9d43902801a95bd. Latest consensus state: current_term: 1 leader_uuid: "0f7f74261aa04615a9d43902801a95bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7f74261aa04615a9d43902801a95bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41199 } } }
I20260928 09:04:34.835347 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:34.835570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 2791
I20260928 09:04:34.835644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:34.835829 392 external_mini_cluster.cc:1428] Setting key 6a99ee41e18a697a374481eb071f827b
I20260928 09:04:34.836123 2866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:34.836860 2866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:34.838887 2866 catalog_manager.cc:1347] Generated new cluster ID: eb65a682ce764f1c99a46581aa00216c
I20260928 09:04:34.838961 2866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:34.834551 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 09:04:34 dist-test-slave-smns krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:34.850859 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:34.836305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.0.98.62:41199)
0928 09:04:34.836444 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:34.836454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:34.836522 (+ 68us) server_negotiation.cc:407] Connection header received
0928 09:04:34.836707 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:34.836715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:34.836769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:34.838281 (+ 1512us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:34.838402 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:34.841775 (+ 3373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:34.843146 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:34.843952 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:34.844368 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:34.846915 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:34.846919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:34.846919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:34.846931 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:34.848913 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:34.849295 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:34.849297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:34.849298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:34.849391 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:34.849604 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:34.849604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:34.849605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:34.849894 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:34.850431 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:34.850499 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 09:04:34.850627 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260928 09:04:34.852387 2866 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:34.853050 2866 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:34.856915 2866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f7f74261aa04615a9d43902801a95bd: Generated new TSK 0
I20260928 09:04:34.857095 2866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:34.885281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41199
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:34.967774 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:34.968016 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:34.968056 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:34.969282 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:34.969357 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:34.969403 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:34.970990 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:34.971290 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:34.971524 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:34.976660 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:34.976665 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:34.976805 2885 server_base.cc:1034] running on GCE node
W20260928 09:04:34.976820 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:34.977159 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:34.977443 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:34.978605 2885 hybrid_clock.cc:648] HybridClock initialized: now 1790586274978616 us; error 53 us; skew 500 ppm
Sep 28 09:04:34 dist-test-slave-smns krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:34.981056 2885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:34.981510 2885 webserver.cc:460] Webserver started at http://127.0.98.1:33247/ using document root <none> and password file <none>
I20260928 09:04:34.981762 2885 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:34.981808 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:34.981899 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:34.982903 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "44055d78901049cbb2c1686553ef6cbc"
format_stamp: "Formatted at 2026-09-28 09:04:34 on dist-test-slave-smns"
server_key: "3aec6a6222ba5380429d11a5d1f8786e"
server_key_iv: "c6431a6d6719b88b31eda426aa756568"
server_key_version: "encryptionkey@0"
I20260928 09:04:34.983210 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44055d78901049cbb2c1686553ef6cbc"
format_stamp: "Formatted at 2026-09-28 09:04:34 on dist-test-slave-smns"
server_key: "3aec6a6222ba5380429d11a5d1f8786e"
server_key_iv: "c6431a6d6719b88b31eda426aa756568"
server_key_version: "encryptionkey@0"
I20260928 09:04:34.984469 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:04:34.985106 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:34.985273 2885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:34.985359 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "44055d78901049cbb2c1686553ef6cbc"
format_stamp: "Formatted at 2026-09-28 09:04:34 on dist-test-slave-smns"
server_key: "3aec6a6222ba5380429d11a5d1f8786e"
server_key_iv: "c6431a6d6719b88b31eda426aa756568"
server_key_version: "encryptionkey@0"
I20260928 09:04:34.985437 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:34.999676 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:35.002117 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:35.002224 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:35.002406 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:35.002656 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:35.002686 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.002714 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:35.002727 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.008459 2885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35079
I20260928 09:04:35.008561 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35079 every 8 connection(s)
I20260928 09:04:35.008862 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:35.010376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 2885
I20260928 09:04:35.010489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:35.010699 392 external_mini_cluster.cc:1428] Setting key 10c64048089079aa68b73b8ffbd25244
Sep 28 09:04:35 dist-test-slave-smns krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 09:04:35.019526 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:35.009605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44681 (local address 127.0.98.62:41199)
0928 09:04:35.009670 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:35.009672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:35.009873 (+ 201us) server_negotiation.cc:407] Connection header received
0928 09:04:35.011146 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:35.011148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:35.011160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:35.011229 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:35.012112 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.012620 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.013807 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.014059 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.014141 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:35.016577 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:35.016579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:35.016579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:35.016588 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:35.018112 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:35.018552 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:35.018553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:35.018553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:35.018616 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:35.018867 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:35.018868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:35.018869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:35.018993 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:35.019069 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:35.019338 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 09:04:35.019423 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:04:35.019748 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:35.009586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41199 (local address 127.0.98.1:44681)
0928 09:04:35.009745 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:35.009788 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:35.011035 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:35.011284 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:35.011288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:35.011728 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:35.012013 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:35.012020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.012709 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.012710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:35.013715 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:35.013717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.014148 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.014149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:35.014280 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:35.014842 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:35.014850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:35.016476 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:35.018220 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:35.018226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:35.018229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:35.018458 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:35.018678 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:35.018679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:35.018680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:35.018792 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:35.019060 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:35.019066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:35.019207 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 09:04:35.019318 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 09:04:35.019479 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260928 09:04:35.020227 3014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41199
I20260928 09:04:35.020337 3014 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:35.020535 3014 heartbeater.cc:511] Master 127.0.98.62:41199 requested a full tablet report, sending...
I20260928 09:04:35.021032 2823 ts_manager.cc:194] Registered new tserver with Master: 44055d78901049cbb2c1686553ef6cbc (127.0.98.1:35079)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 09:04:35.021845 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:35.037082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41199
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:35.119650 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:35.119868 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:35.119892 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:35.121097 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:35.121143 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:35.121171 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:35.122854 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:35.123116 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:35.123366 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:35.128728 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:35.128803 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:35.128728 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:35.128757 3019 server_base.cc:1034] running on GCE node
I20260928 09:04:35.129096 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:35.129305 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:35.130441 3019 hybrid_clock.cc:648] HybridClock initialized: now 1790586275130430 us; error 30 us; skew 500 ppm
Sep 28 09:04:35 dist-test-slave-smns krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:35.132886 3019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:35.133358 3019 webserver.cc:460] Webserver started at http://127.0.98.2:36607/ using document root <none> and password file <none>
I20260928 09:04:35.133643 3019 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:35.133692 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:35.133821 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:35.134856 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "361b714d53be4374a05c338745170c70"
format_stamp: "Formatted at 2026-09-28 09:04:35 on dist-test-slave-smns"
server_key: "344ae593348708fcc0415aa3d6bbc055"
server_key_iv: "bb33c884ca5d02fc9233232c28e47c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:35.135218 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "361b714d53be4374a05c338745170c70"
format_stamp: "Formatted at 2026-09-28 09:04:35 on dist-test-slave-smns"
server_key: "344ae593348708fcc0415aa3d6bbc055"
server_key_iv: "bb33c884ca5d02fc9233232c28e47c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:35.136523 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:35.137321 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.137518 3019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:35.137619 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "361b714d53be4374a05c338745170c70"
format_stamp: "Formatted at 2026-09-28 09:04:35 on dist-test-slave-smns"
server_key: "344ae593348708fcc0415aa3d6bbc055"
server_key_iv: "bb33c884ca5d02fc9233232c28e47c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:35.137684 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:35.158915 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:35.161239 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:35.161350 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:35.161598 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:35.161900 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:35.161931 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.161955 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:35.161967 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.167625 3019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37833
I20260928 09:04:35.167703 3147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37833 every 8 connection(s)
I20260928 09:04:35.168026 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:35.172271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3019
I20260928 09:04:35.172366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:35.172570 392 external_mini_cluster.cc:1428] Setting key 1e60cfb91ead22d6ea6b7089fc91ea7f
Sep 28 09:04:35 dist-test-slave-smns krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:35.178238 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:35.168688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41199 (local address 127.0.98.2:53575)
0928 09:04:35.168855 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:35.168907 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:35.169946 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:35.170219 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:35.170223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:35.170469 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:35.170656 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:35.170660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.171288 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.171289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:35.172070 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:35.172072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.172564 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.172565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:35.172715 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:35.173381 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:35.173390 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:35.174859 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:35.176759 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:35.176767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:35.176770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:35.177088 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:35.177318 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:35.177318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:35.177319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:35.177417 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:35.177756 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:35.177760 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:35.177875 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 09:04:35.177963 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:04:35.178081 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260928 09:04:35.178248 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:35.168734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53575 (local address 127.0.98.62:41199)
0928 09:04:35.168802 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:35.168808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:35.168995 (+ 187us) server_negotiation.cc:407] Connection header received
0928 09:04:35.170037 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:35.170038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:35.170047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:35.170112 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:35.170725 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.171212 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.172153 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.172459 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.172540 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:35.174955 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:35.174957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:35.174958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:35.174966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:35.176651 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:35.177175 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:35.177178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:35.177178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:35.177242 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:35.177490 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:35.177492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:35.177493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:35.177680 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:35.177758 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:35.178014 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 09:04:35.178109 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:04:35.178722 3148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41199
I20260928 09:04:35.178829 3148 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:35.179021 3148 heartbeater.cc:511] Master 127.0.98.62:41199 requested a full tablet report, sending...
I20260928 09:04:35.179384 2823 ts_manager.cc:194] Registered new tserver with Master: 361b714d53be4374a05c338745170c70 (127.0.98.2:37833)
I20260928 09:04:35.179777 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53575
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:35.202111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41199
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:35.282501 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:35.282759 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:35.282804 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:35.284053 3153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:35.284118 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:35.284164 3153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:35.285723 3153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:35.286015 3153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:35.286245 3153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:35.291579 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:35.291620 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:35.291697 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:35.291824 3153 server_base.cc:1034] running on GCE node
I20260928 09:04:35.292070 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:35.292349 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:35.293504 3153 hybrid_clock.cc:648] HybridClock initialized: now 1790586275293483 us; error 38 us; skew 500 ppm
Sep 28 09:04:35 dist-test-slave-smns krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:35.296240 3153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:35.296696 3153 webserver.cc:460] Webserver started at http://127.0.98.3:38563/ using document root <none> and password file <none>
I20260928 09:04:35.296963 3153 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:35.297041 3153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:35.297189 3153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:35.298312 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "7765f4ff459d430fbc6eb0f97f37c25d"
format_stamp: "Formatted at 2026-09-28 09:04:35 on dist-test-slave-smns"
server_key: "2efeb80195e8621bad81be4686411dbc"
server_key_iv: "584fcc655d68db7fdf5c39f288250d60"
server_key_version: "encryptionkey@0"
I20260928 09:04:35.298689 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7765f4ff459d430fbc6eb0f97f37c25d"
format_stamp: "Formatted at 2026-09-28 09:04:35 on dist-test-slave-smns"
server_key: "2efeb80195e8621bad81be4686411dbc"
server_key_iv: "584fcc655d68db7fdf5c39f288250d60"
server_key_version: "encryptionkey@0"
I20260928 09:04:35.300048 3153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:35.300951 3168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.301128 3153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:35.301226 3153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "7765f4ff459d430fbc6eb0f97f37c25d"
format_stamp: "Formatted at 2026-09-28 09:04:35 on dist-test-slave-smns"
server_key: "2efeb80195e8621bad81be4686411dbc"
server_key_iv: "584fcc655d68db7fdf5c39f288250d60"
server_key_version: "encryptionkey@0"
I20260928 09:04:35.301337 3153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:35.323661 3153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:35.326268 3153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:35.326447 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:35.326690 3153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:35.327049 3153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:35.327106 3153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.327147 3153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:35.327174 3153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:35.333182 3153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38089
I20260928 09:04:35.333238 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38089 every 8 connection(s)
I20260928 09:04:35.333602 3153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:35.336853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3153
I20260928 09:04:35.336982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:35.337190 392 external_mini_cluster.cc:1428] Setting key 04d4922bbfc2483187ab946cac6b3796
Sep 28 09:04:35 dist-test-slave-smns krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:35.343910 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:35.334358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34895 (local address 127.0.98.62:41199)
0928 09:04:35.334411 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:35.334413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:35.334590 (+ 177us) server_negotiation.cc:407] Connection header received
0928 09:04:35.335615 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:35.335617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:35.335631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:35.335692 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:35.336373 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.336894 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.338193 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.338404 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.338490 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:35.340814 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:35.340816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:35.340816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:35.340823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:35.342443 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:35.342873 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:35.342875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:35.342875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:35.342934 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:35.343201 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:35.343202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:35.343203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:35.343329 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:35.343433 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:35.343722 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 09:04:35.343819 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:04:35.344101 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:35.334307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41199 (local address 127.0.98.3:34895)
0928 09:04:35.334454 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:35.334501 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:35.335489 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:35.335772 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:35.335776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:35.336065 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:35.336258 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:35.336262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.336998 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.336999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:35.338096 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:35.338098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:35.338495 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:35.338496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:35.338633 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:35.339248 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:35.339263 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:35.340703 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:35.342542 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:35.342549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:35.342551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:35.342778 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:35.343007 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:35.343008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:35.343009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:35.343130 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:35.343426 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:35.343433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:35.343587 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 09:04:35.343700 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:04:35.343859 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260928 09:04:35.344604 3282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41199
I20260928 09:04:35.344697 3282 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:35.344846 3282 heartbeater.cc:511] Master 127.0.98.62:41199 requested a full tablet report, sending...
I20260928 09:04:35.345201 2823 ts_manager.cc:194] Registered new tserver with Master: 7765f4ff459d430fbc6eb0f97f37c25d (127.0.98.3:38089)
I20260928 09:04:35.345652 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34895
I20260928 09:04:35.349053 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 09:04:35.360123 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:35.359172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.98.62:41199)
0928 09:04:35.359236 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:35.359238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:35.359372 (+ 134us) server_negotiation.cc:407] Connection header received
0928 09:04:35.359425 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:35.359426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:35.359435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:35.359487 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:35.360050 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57502: BlockingRecv error: recv got EOF from 127.0.0.1:57502 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:04:35.360901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 2885
I20260928 09:04:35.365231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3019
I20260928 09:04:35.369671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3153
I20260928 09:04:35.373910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 2791
2026-09-28T09:04:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1843 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:36 dist-test-slave-smns krb5kdc[3294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:36 dist-test-slave-smns krb5kdc[3294](info): set up 2 sockets
Sep 28 09:04:36 dist-test-slave-smns krb5kdc[3294](info): commencing operation
krb5kdc: starting...
W20260928 09:04:37.897482 392 mini_kdc.cc:121] Time spent starting KDC: real 2.501s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:37 dist-test-slave-smns krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:38.018528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:38.098742 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:38.098946 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:38.098968 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:38.100126 3307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:38.100167 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:38.100180 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:38.100193 3307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:38.100204 3307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:38.101703 3307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:38.101966 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:38.102219 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:38.107141 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:38.107167 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:38.107215 3307 server_base.cc:1034] running on GCE node
W20260928 09:04:38.107185 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:38.107530 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:38.107761 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:38.108981 3307 hybrid_clock.cc:648] HybridClock initialized: now 1790586278108944 us; error 46 us; skew 500 ppm
Sep 28 09:04:38 dist-test-slave-smns krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:38.111598 3307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:38.112067 3307 webserver.cc:460] Webserver started at http://127.0.98.62:42117/ using document root <none> and password file <none>
I20260928 09:04:38.112308 3307 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:38.112385 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:38.112534 3307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:38.113678 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "3b4e3b65480341ce8e676406ce0a68f6"
format_stamp: "Formatted at 2026-09-28 09:04:38 on dist-test-slave-smns"
server_key: "9f47b9e5b475951ef301200230dcf408"
server_key_iv: "c3907d44fddd6d7989568adf958d2dbc"
server_key_version: "encryptionkey@0"
I20260928 09:04:38.114087 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b4e3b65480341ce8e676406ce0a68f6"
format_stamp: "Formatted at 2026-09-28 09:04:38 on dist-test-slave-smns"
server_key: "9f47b9e5b475951ef301200230dcf408"
server_key_iv: "c3907d44fddd6d7989568adf958d2dbc"
server_key_version: "encryptionkey@0"
I20260928 09:04:38.115412 3307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:38.116161 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:38.116330 3307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:38.116465 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "3b4e3b65480341ce8e676406ce0a68f6"
format_stamp: "Formatted at 2026-09-28 09:04:38 on dist-test-slave-smns"
server_key: "9f47b9e5b475951ef301200230dcf408"
server_key_iv: "c3907d44fddd6d7989568adf958d2dbc"
server_key_version: "encryptionkey@0"
I20260928 09:04:38.116542 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:38.126205 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:38.128461 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:38.128618 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:38.132618 3307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34031
I20260928 09:04:38.132663 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34031 every 8 connection(s)
I20260928 09:04:38.133016 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:38.133971 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:38.136169 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6: Bootstrap starting.
I20260928 09:04:38.136755 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:38.137066 3375 log.cc:824] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:38.137811 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6: No bootstrap required, opened a new log
I20260928 09:04:38.139353 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4e3b65480341ce8e676406ce0a68f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34031 } }
I20260928 09:04:38.139489 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:38.139513 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4e3b65480341ce8e676406ce0a68f6, State: Initialized, Role: FOLLOWER
I20260928 09:04:38.139690 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4e3b65480341ce8e676406ce0a68f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34031 } }
I20260928 09:04:38.139752 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:38.139794 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:38.139842 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:38.140601 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4e3b65480341ce8e676406ce0a68f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34031 } }
I20260928 09:04:38.140686 3375 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4e3b65480341ce8e676406ce0a68f6; no voters:
I20260928 09:04:38.140856 3375 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:38.140899 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:38.141122 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4e3b65480341ce8e676406ce0a68f6, State: Running, Role: LEADER
I20260928 09:04:38.141155 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:38.141213 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4e3b65480341ce8e676406ce0a68f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34031 } }
I20260928 09:04:38.141623 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4e3b65480341ce8e676406ce0a68f6. Latest consensus state: current_term: 1 leader_uuid: "3b4e3b65480341ce8e676406ce0a68f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4e3b65480341ce8e676406ce0a68f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34031 } } }
I20260928 09:04:38.141619 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4e3b65480341ce8e676406ce0a68f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4e3b65480341ce8e676406ce0a68f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34031 } } }
I20260928 09:04:38.141847 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:38.142059 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:38.142302 3387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:38.142782 3387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:38.143098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3307
I20260928 09:04:38.143186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:38.143354 392 external_mini_cluster.cc:1428] Setting key b56d93cf9e5fbf34d92b0a281af6de22
I20260928 09:04:38.144500 3387 catalog_manager.cc:1347] Generated new cluster ID: ae122395e5724afeafc0760f4a029711
I20260928 09:04:38.144567 3387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:04:38 dist-test-slave-smns krb5kdc[3294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:38.155488 3387 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:38.156222 3387 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:38.157189 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:38.143967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.0.98.62:34031)
0928 09:04:38.145723 (+ 1756us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:38.145732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:38.145812 (+ 80us) server_negotiation.cc:407] Connection header received
0928 09:04:38.145929 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:38.145935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:38.145991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:38.146121 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:38.146944 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:38.148715 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:38.149840 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:38.150480 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:38.150584 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:38.153414 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:38.153419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:38.153419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:38.153453 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:38.155477 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:38.155892 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:38.155895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:38.155895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:38.156002 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:38.156223 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:38.156224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:38.156224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:38.156547 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:38.156700 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:38.156819 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 09:04:38.156962 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1711,"thread_start_us":81,"threads_started":1}
I20260928 09:04:38.158134 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:04:38.163357 3387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4e3b65480341ce8e676406ce0a68f6: Generated new TSK 0
I20260928 09:04:38.163535 3387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:04:38.182379 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:38.178482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.0.98.62:34031)
0928 09:04:38.178549 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:38.178551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:38.178693 (+ 142us) server_negotiation.cc:407] Connection header received
0928 09:04:38.178760 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:38.178761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:38.178773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:38.178830 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:38.179204 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:38.180027 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:38.180749 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:38.181101 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:38.181171 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:38.181859 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:38.181860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:38.181861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:38.182113 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:38.182131 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:38.182138 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:38.182200 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:38.182269 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 09:04:38.182294 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:04:38.184762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3307
2026-09-28T09:04:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2806 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:38 dist-test-slave-smns krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:38 dist-test-slave-smns krb5kdc[3408](info): set up 2 sockets
Sep 28 09:04:38 dist-test-slave-smns krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260928 09:04:39.233837 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:39.352805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42519
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:39.436622 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:39.436856 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:39.436892 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:39.438052 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:39.438117 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:39.438143 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:39.438170 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:39.438197 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:39.439630 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:39.439910 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:39.440133 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:39.445330 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:39.445312 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:39.445341 3421 server_base.cc:1034] running on GCE node
W20260928 09:04:39.445312 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:39.445782 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:39.446038 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:39.447178 3421 hybrid_clock.cc:648] HybridClock initialized: now 1790586279447158 us; error 38 us; skew 500 ppm
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:39.449654 3421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:39.450075 3421 webserver.cc:460] Webserver started at http://127.0.98.62:36365/ using document root <none> and password file <none>
I20260928 09:04:39.450348 3421 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:39.450394 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:39.450541 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:39.451603 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "5829a2ccc3244b58a9f0f73fc82179d6"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "884ef1e0d76105870bb3b5bdcf8855d1"
server_key_iv: "49f500e2381618cc976e2883a0083a62"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.451954 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "5829a2ccc3244b58a9f0f73fc82179d6"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "884ef1e0d76105870bb3b5bdcf8855d1"
server_key_iv: "49f500e2381618cc976e2883a0083a62"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.453224 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:04:39.454053 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.454262 3421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:04:39.454350 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "5829a2ccc3244b58a9f0f73fc82179d6"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "884ef1e0d76105870bb3b5bdcf8855d1"
server_key_iv: "49f500e2381618cc976e2883a0083a62"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.454421 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:39.472259 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:39.474476 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:39.474637 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:39.478653 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42519
I20260928 09:04:39.478682 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42519 every 8 connection(s)
I20260928 09:04:39.479027 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:39.479997 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:39.482324 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6: Bootstrap starting.
I20260928 09:04:39.482896 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:39.483175 3489 log.cc:824] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:39.483826 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6: No bootstrap required, opened a new log
I20260928 09:04:39.485419 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42519 } }
I20260928 09:04:39.485598 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:39.485664 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5829a2ccc3244b58a9f0f73fc82179d6, State: Initialized, Role: FOLLOWER
I20260928 09:04:39.485822 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42519 } }
I20260928 09:04:39.485908 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:39.485930 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:39.485958 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:39.486768 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42519 } }
I20260928 09:04:39.486852 3489 leader_election.cc:302] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5829a2ccc3244b58a9f0f73fc82179d6; no voters:
I20260928 09:04:39.487010 3489 leader_election.cc:288] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:39.487051 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:39.487268 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:39.487278 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5829a2ccc3244b58a9f0f73fc82179d6, State: Running, Role: LEADER
I20260928 09:04:39.487511 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42519 } }
I20260928 09:04:39.487579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3421
I20260928 09:04:39.487700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:39.487913 392 external_mini_cluster.cc:1428] Setting key a264dbcafd4b2fad21999f97e5a27ffb
I20260928 09:04:39.487879 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42519 } } }
I20260928 09:04:39.487953 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:39.488116 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5829a2ccc3244b58a9f0f73fc82179d6. Latest consensus state: current_term: 1 leader_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5829a2ccc3244b58a9f0f73fc82179d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42519 } } }
I20260928 09:04:39.488202 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:39.488631 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:39.489133 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:39.493979 3501 catalog_manager.cc:1347] Generated new cluster ID: 074f45a95a1c4d1192ca662a48712f26
I20260928 09:04:39.494040 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:39.499253 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:39.488429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.0.98.62:42519)
0928 09:04:39.488951 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:39.488960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:39.488991 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:04:39.489109 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:39.489116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:39.489174 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:39.489328 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:39.490069 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.491396 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.492168 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.492820 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.492954 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:39.495457 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:39.495461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:39.495462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:39.495475 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:39.497709 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:39.498084 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:39.498087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:39.498088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:39.498194 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:39.498400 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:39.498401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:39.498402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:39.498648 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:39.498809 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:39.498919 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 09:04:39.499078 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":94,"threads_started":1}
I20260928 09:04:39.504490 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:39.505003 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:39.507088 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5829a2ccc3244b58a9f0f73fc82179d6: Generated new TSK 0
I20260928 09:04:39.507246 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:39.538064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42519
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:39.641427 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:39.641709 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:39.641750 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:39.643005 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:39.643069 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:39.643115 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:39.644577 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:39.644851 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:39.645092 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:39.650431 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:39.650431 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:39.650427 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:39.650925 3515 server_base.cc:1034] running on GCE node
I20260928 09:04:39.651147 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:39.651407 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:39.652606 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790586279652587 us; error 34 us; skew 500 ppm
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:39.656049 3515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:39.656685 3515 webserver.cc:460] Webserver started at http://127.0.98.1:41265/ using document root <none> and password file <none>
I20260928 09:04:39.657069 3515 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:39.657145 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:39.657281 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:39.658962 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "0f3e2452603848fda8170e5f0d8bdfb3"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "75f7077f97303bf18f59a437590b964d"
server_key_iv: "32cccfbd82891a8c48fed3ff93d45cf7"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.659415 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0f3e2452603848fda8170e5f0d8bdfb3"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "75f7077f97303bf18f59a437590b964d"
server_key_iv: "32cccfbd82891a8c48fed3ff93d45cf7"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.661216 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:39.663065 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.663256 3515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:04:39.663347 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "0f3e2452603848fda8170e5f0d8bdfb3"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "75f7077f97303bf18f59a437590b964d"
server_key_iv: "32cccfbd82891a8c48fed3ff93d45cf7"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.663426 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:39.691794 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:39.694862 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:39.695034 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:39.695372 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:39.695789 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:39.695843 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.695879 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:39.695910 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.703926 3515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35003
I20260928 09:04:39.704008 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35003 every 8 connection(s)
I20260928 09:04:39.704367 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:39.714033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3515
I20260928 09:04:39.714125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:39.714136 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:39.705144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38947 (local address 127.0.98.62:42519)
0928 09:04:39.705203 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:39.705205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:39.705444 (+ 239us) server_negotiation.cc:407] Connection header received
0928 09:04:39.706097 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:39.706099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:39.706115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:39.706173 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:39.707167 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.707689 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.708741 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.708976 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.709072 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:39.711228 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:39.711230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:39.711230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:39.711236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:39.712762 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:39.713149 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:39.713150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:39.713150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:39.713198 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:39.713433 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:39.713434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:39.713434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:39.713536 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:39.713652 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:39.713925 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 09:04:39.714019 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:04:39.714290 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:39.705120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42519 (local address 127.0.98.1:38947)
0928 09:04:39.705314 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:39.705352 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:39.705972 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:39.706257 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:39.706260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:39.706855 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:39.707053 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:39.707058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.707798 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.707800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:39.708624 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:39.708626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.709081 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.709082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:39.709209 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:39.709772 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:39.709780 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:39.711141 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:39.712847 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:39.712853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:39.712855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:39.713077 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:39.713261 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:39.713261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:39.713262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:39.713355 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:39.713661 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:39.713665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:39.713783 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 09:04:39.713873 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 09:04:39.714025 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260928 09:04:39.714361 392 external_mini_cluster.cc:1428] Setting key 5fdd2d55bd1a11dba5738e1d7321bc67
I20260928 09:04:39.714843 3644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42519
I20260928 09:04:39.714954 3644 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:39.715138 3644 heartbeater.cc:511] Master 127.0.98.62:42519 requested a full tablet report, sending...
I20260928 09:04:39.715700 3453 ts_manager.cc:194] Registered new tserver with Master: 0f3e2452603848fda8170e5f0d8bdfb3 (127.0.98.1:35003)
I20260928 09:04:39.716315 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38947
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:39.740836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42519
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:39.823694 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:39.823889 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:39.823913 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:39.825081 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:39.825119 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:39.825145 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:39.826588 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:39.826789 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:39.826957 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:39.831987 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:39.832010 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:39.831996 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:39.832098 3649 server_base.cc:1034] running on GCE node
I20260928 09:04:39.832413 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:39.832617 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:39.833758 3649 hybrid_clock.cc:648] HybridClock initialized: now 1790586279833735 us; error 37 us; skew 500 ppm
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:39.844473 3649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:39.844995 3649 webserver.cc:460] Webserver started at http://127.0.98.2:35081/ using document root <none> and password file <none>
I20260928 09:04:39.845261 3649 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:39.845328 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:39.845467 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:39.846735 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ad1147dbf914063936313adc21adcb1"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "fe67bc37a908553ae568881da2b7ae8e"
server_key_iv: "1d5f32dff31dda93f9b2594b71c2e4ff"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.847133 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ad1147dbf914063936313adc21adcb1"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "fe67bc37a908553ae568881da2b7ae8e"
server_key_iv: "1d5f32dff31dda93f9b2594b71c2e4ff"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.848492 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:39.849226 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.849402 3649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:39.849507 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "1ad1147dbf914063936313adc21adcb1"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "fe67bc37a908553ae568881da2b7ae8e"
server_key_iv: "1d5f32dff31dda93f9b2594b71c2e4ff"
server_key_version: "encryptionkey@0"
I20260928 09:04:39.849632 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:39.862490 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:39.864446 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:39.864616 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:39.864856 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:39.865182 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:39.865242 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.865284 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:39.865312 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:39.871547 3649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39587
I20260928 09:04:39.871589 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39587 every 8 connection(s)
I20260928 09:04:39.871913 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:39.875825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3649
I20260928 09:04:39.875936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:39.876125 392 external_mini_cluster.cc:1428] Setting key d44d961d83227f10cf42a237889d84a4
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:39.881839 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:39.872699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42181 (local address 127.0.98.62:42519)
0928 09:04:39.872746 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:39.872747 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:39.873013 (+ 266us) server_negotiation.cc:407] Connection header received
0928 09:04:39.873449 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:39.873450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:39.873461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:39.873524 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:39.874522 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.875023 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.876281 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.876540 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.876634 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:39.878933 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:39.878935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:39.878935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:39.878942 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:39.880386 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:39.880825 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:39.880827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:39.880827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:39.880893 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:39.881158 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:39.881159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:39.881159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:39.881275 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:39.881367 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:39.881661 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 09:04:39.881744 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:04:39.882067 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:39.872648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42519 (local address 127.0.98.2:42181)
0928 09:04:39.872810 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:39.872851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:39.873334 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:39.873642 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:39.873647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:39.874210 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:39.874398 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:39.874403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.875112 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.875112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:39.876156 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:39.876160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:39.876639 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:39.876640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:39.876811 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:39.877409 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:39.877422 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:39.878829 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:39.880496 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:39.880503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:39.880505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:39.880727 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:39.880966 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:39.880967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:39.880968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:39.881086 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:39.881358 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:39.881365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:39.881504 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 09:04:39.881641 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 09:04:39.881819 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260928 09:04:39.882584 3778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42519
I20260928 09:04:39.882676 3778 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:39.882872 3778 heartbeater.cc:511] Master 127.0.98.62:42519 requested a full tablet report, sending...
I20260928 09:04:39.883271 3453 ts_manager.cc:194] Registered new tserver with Master: 1ad1147dbf914063936313adc21adcb1 (127.0.98.2:39587)
I20260928 09:04:39.883663 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42181
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:39.904218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42519
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:39.985525 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:39.985808 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:39.985832 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:39.987118 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:39.987159 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:39.987185 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:39.988852 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:39.989117 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:39.989352 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:39.994586 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:39.994673 3783 server_base.cc:1034] running on GCE node
W20260928 09:04:39.994652 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:39.994561 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:39.994973 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:39.995242 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:39.996407 3783 hybrid_clock.cc:648] HybridClock initialized: now 1790586279996395 us; error 31 us; skew 500 ppm
Sep 28 09:04:39 dist-test-slave-smns krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:39.999100 3783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:39.999630 3783 webserver.cc:460] Webserver started at http://127.0.98.3:34603/ using document root <none> and password file <none>
I20260928 09:04:39.999887 3783 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:39.999962 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:40.000113 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:40.001371 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "23bb10e43dcc4997a02d877137ca25a5"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "67ae394326a068cafb2c0052b886e5ba"
server_key_iv: "7f1fd8b4e6d17df52d3de8bf14d1312c"
server_key_version: "encryptionkey@0"
I20260928 09:04:40.001842 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23bb10e43dcc4997a02d877137ca25a5"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "67ae394326a068cafb2c0052b886e5ba"
server_key_iv: "7f1fd8b4e6d17df52d3de8bf14d1312c"
server_key_version: "encryptionkey@0"
I20260928 09:04:40.003198 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:40.004087 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:40.004326 3783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:40.004436 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "23bb10e43dcc4997a02d877137ca25a5"
format_stamp: "Formatted at 2026-09-28 09:04:39 on dist-test-slave-smns"
server_key: "67ae394326a068cafb2c0052b886e5ba"
server_key_iv: "7f1fd8b4e6d17df52d3de8bf14d1312c"
server_key_version: "encryptionkey@0"
I20260928 09:04:40.004531 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:40.022053 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:40.024078 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:40.024246 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:40.024483 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:40.024802 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:40.024861 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:40.024905 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:40.024931 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:40.031292 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40801
I20260928 09:04:40.031342 3911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40801 every 8 connection(s)
I20260928 09:04:40.031684 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:40 dist-test-slave-smns krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:40.039111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3783
I20260928 09:04:40.039222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:40.039431 392 external_mini_cluster.cc:1428] Setting key 4d8413690c8a42e0d1062a7892accf90
I20260928 09:04:40.041406 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:40.032480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39237 (local address 127.0.98.62:42519)
0928 09:04:40.032552 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:40.032554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:40.032738 (+ 184us) server_negotiation.cc:407] Connection header received
0928 09:04:40.033221 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:40.033222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:40.033232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:40.033299 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:40.034307 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:40.034798 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:40.035874 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:40.036060 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:40.036116 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:40.038188 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:40.038189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:40.038190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:40.038197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:40.039856 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:40.040314 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:40.040316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:40.040316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:40.040384 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:40.040695 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:40.040696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:40.040697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:40.040828 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:40.040919 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:40.041222 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 09:04:40.041309 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 09:04:40.041639 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:40.032454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42519 (local address 127.0.98.3:39237)
0928 09:04:40.032604 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:40.032648 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:40.033127 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:40.033357 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:40.033360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:40.034020 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:40.034217 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:40.034222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:40.034945 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:40.034945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:40.035747 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:40.035749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:40.036138 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:40.036139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:40.036233 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:40.036686 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:40.036694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:40.038084 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:40.039954 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:40.039961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:40.039964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:40.040211 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:40.040457 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:40.040458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:40.040458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:40.040603 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:40.040928 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:40.040935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:40.041074 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 09:04:40.041191 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 09:04:40.041365 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260928 09:04:40.042164 3912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42519
I20260928 09:04:40.042281 3912 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:40.042444 3912 heartbeater.cc:511] Master 127.0.98.62:42519 requested a full tablet report, sending...
I20260928 09:04:40.042770 3453 ts_manager.cc:194] Registered new tserver with Master: 23bb10e43dcc4997a02d877137ca25a5 (127.0.98.3:40801)
I20260928 09:04:40.043105 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39237
I20260928 09:04:40.051244 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 09:04:40 dist-test-slave-smns krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586280, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 09:04:40 dist-test-slave-smns krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586280, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:40.074749 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:40.067393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.0.98.62:42519)
0928 09:04:40.067544 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:40.067550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:40.067564 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:04:40.067643 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:40.067644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:40.067653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:40.067714 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:40.068269 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:40.068903 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:40.069675 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:40.069910 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:40.069985 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:40.072112 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:40.072113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:40.072114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:40.072120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:40.073607 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:40.073933 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:40.073935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:40.073935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:40.073994 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:40.074261 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:40.074262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:40.074262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:40.074425 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:40.074513 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:40.074612 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 09:04:40.074676 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260928 09:04:40.075109 3453 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36922
I20260928 09:04:40.075623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3515
I20260928 09:04:40.081104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3649
I20260928 09:04:40.085809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3783
I20260928 09:04:40.090404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3421
2026-09-28T09:04:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1907 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:40 dist-test-slave-smns krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:40 dist-test-slave-smns krb5kdc[3924](info): set up 2 sockets
Sep 28 09:04:40 dist-test-slave-smns krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260928 09:04:41.137988 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:41.265831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:41.347244 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:41.347512 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:41.347548 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:41.348742 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:41.348825 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:41.348850 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:41.348873 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:41.348899 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:41.350369 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:41.350652 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:41.350881 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:41.355981 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.356074 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.355948 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:41.356204 3937 server_base.cc:1034] running on GCE node
I20260928 09:04:41.356492 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:41.356777 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:41.357941 3937 hybrid_clock.cc:648] HybridClock initialized: now 1790586281357925 us; error 52 us; skew 500 ppm
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:41.360378 3937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:41.360826 3937 webserver.cc:460] Webserver started at http://127.0.98.62:41511/ using document root <none> and password file <none>
I20260928 09:04:41.361063 3937 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:41.361136 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:41.361285 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:41.362414 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "260106f9a6e14eda930b77f303c8258c"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "19505dc1cd1e5fdb426329f916354ad2"
server_key_iv: "a9fafa7bb1200a954c50a5f2b2c5d17e"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.362782 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "260106f9a6e14eda930b77f303c8258c"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "19505dc1cd1e5fdb426329f916354ad2"
server_key_iv: "a9fafa7bb1200a954c50a5f2b2c5d17e"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.364283 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:04:41.365263 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.365484 3937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:41.365624 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "260106f9a6e14eda930b77f303c8258c"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "19505dc1cd1e5fdb426329f916354ad2"
server_key_iv: "a9fafa7bb1200a954c50a5f2b2c5d17e"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.365711 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:41.394057 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:41.396026 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:41.396200 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:41.400046 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39821
I20260928 09:04:41.400084 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39821 every 8 connection(s)
I20260928 09:04:41.400493 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:41.401419 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:41.403682 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c: Bootstrap starting.
I20260928 09:04:41.404198 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:41.404428 4005 log.cc:824] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:41.405189 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c: No bootstrap required, opened a new log
I20260928 09:04:41.406737 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260106f9a6e14eda930b77f303c8258c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39821 } }
I20260928 09:04:41.406905 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:41.406931 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260106f9a6e14eda930b77f303c8258c, State: Initialized, Role: FOLLOWER
I20260928 09:04:41.407078 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260106f9a6e14eda930b77f303c8258c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39821 } }
I20260928 09:04:41.407159 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:41.407183 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:41.407212 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:41.407703 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260106f9a6e14eda930b77f303c8258c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39821 } }
I20260928 09:04:41.407788 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 260106f9a6e14eda930b77f303c8258c; no voters:
I20260928 09:04:41.407953 4005 leader_election.cc:288] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:41.407981 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:41.408239 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:41.408301 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [term 1 LEADER]: Becoming Leader. State: Replica: 260106f9a6e14eda930b77f303c8258c, State: Running, Role: LEADER
I20260928 09:04:41.408444 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260106f9a6e14eda930b77f303c8258c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39821 } }
I20260928 09:04:41.408806 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 260106f9a6e14eda930b77f303c8258c. Latest consensus state: current_term: 1 leader_uuid: "260106f9a6e14eda930b77f303c8258c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260106f9a6e14eda930b77f303c8258c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39821 } } }
I20260928 09:04:41.408815 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "260106f9a6e14eda930b77f303c8258c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260106f9a6e14eda930b77f303c8258c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39821 } } }
I20260928 09:04:41.409085 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:41.408996 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:41.409448 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:41.410012 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:41.410661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 3937
I20260928 09:04:41.410787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:41.411012 392 external_mini_cluster.cc:1428] Setting key 337a77ebe73475f1684903d33c1f60f8
I20260928 09:04:41.411791 4017 catalog_manager.cc:1347] Generated new cluster ID: 34a75a457ee24fb4a2863af8cb89bc75
I20260928 09:04:41.411844 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:41.422483 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.411552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.98.62:39821)
0928 09:04:41.411710 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:41.411719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:41.411750 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:04:41.413834 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:41.413840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:41.413885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:41.414023 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:41.414667 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.415617 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.416385 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.416798 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.416930 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.419284 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:41.419288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:41.419288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:41.419305 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:41.420940 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.421314 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.421317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.421318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.421404 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.421644 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.421645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.421646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.421878 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:41.422069 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:41.422144 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 09:04:41.422289 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 09:04:41.424789 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:41.425469 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:41.428092 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 260106f9a6e14eda930b77f303c8258c: Generated new TSK 0
I20260928 09:04:41.428243 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:41.461221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39821
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:41.542950 4031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:41.543184 4031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:41.543226 4031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:41.544391 4031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:41.544478 4031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:41.544526 4031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:41.546120 4031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:41.546389 4031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:41.546629 4031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:41.551903 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.551915 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.551921 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:41.552285 4031 server_base.cc:1034] running on GCE node
I20260928 09:04:41.552464 4031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:41.552685 4031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:41.553820 4031 hybrid_clock.cc:648] HybridClock initialized: now 1790586281553806 us; error 29 us; skew 500 ppm
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:41.556370 4031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:41.556866 4031 webserver.cc:460] Webserver started at http://127.0.98.1:42759/ using document root <none> and password file <none>
I20260928 09:04:41.557103 4031 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:41.557179 4031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:41.557343 4031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:41.558460 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c6b35fc5a0245a3adc9fbdff5333bf6"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "7c73d281358c0fb85a44320c2f2cafa7"
server_key_iv: "c32d091154b99ae33b8410675b4e0ec2"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.558843 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c6b35fc5a0245a3adc9fbdff5333bf6"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "7c73d281358c0fb85a44320c2f2cafa7"
server_key_iv: "c32d091154b99ae33b8410675b4e0ec2"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.560211 4031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:41.561003 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.561218 4031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:41.561324 4031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "2c6b35fc5a0245a3adc9fbdff5333bf6"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "7c73d281358c0fb85a44320c2f2cafa7"
server_key_iv: "c32d091154b99ae33b8410675b4e0ec2"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.561417 4031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:41.579393 4031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:41.581307 4031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:41.581494 4031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:41.581772 4031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:41.582100 4031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:41.582162 4031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.582198 4031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:41.582226 4031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.588101 4031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45957
I20260928 09:04:41.588138 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45957 every 8 connection(s)
I20260928 09:04:41.588479 4031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:41.596267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4031
I20260928 09:04:41.596385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:41.596586 392 external_mini_cluster.cc:1428] Setting key 5659f8ab1fa62592706e18260506858d
I20260928 09:04:41.598618 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.589290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53941 (local address 127.0.98.62:39821)
0928 09:04:41.589365 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:41.589368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:41.589586 (+ 218us) server_negotiation.cc:407] Connection header received
0928 09:04:41.590586 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:41.590587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:41.590604 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:41.590657 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:41.591308 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.591815 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.592896 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.593108 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.593201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.595346 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:41.595348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:41.595348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:41.595355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:41.596822 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.597624 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.597625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.597625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.597682 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.597923 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.597924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.597924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.598072 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:41.598142 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:41.598380 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 09:04:41.598481 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:04:41.598699 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.589277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39821 (local address 127.0.98.1:53941)
0928 09:04:41.589435 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:41.589470 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:41.590457 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:41.590735 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:41.590738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:41.591008 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:41.591192 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:41.591196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.591943 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.591944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:41.592786 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:41.592788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.593233 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.593234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:41.593357 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.593908 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:41.593917 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:41.595233 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:41.596951 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:41.596958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:41.596960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:41.597257 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:41.597753 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:41.597754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:41.597754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:41.597856 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:41.598137 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:41.598140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:41.598255 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 09:04:41.598335 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:04:41.598454 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260928 09:04:41.599448 4160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39821
I20260928 09:04:41.599553 4160 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:41.599747 4160 heartbeater.cc:511] Master 127.0.98.62:39821 requested a full tablet report, sending...
I20260928 09:04:41.600209 3969 ts_manager.cc:194] Registered new tserver with Master: 2c6b35fc5a0245a3adc9fbdff5333bf6 (127.0.98.1:45957)
I20260928 09:04:41.600800 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53941
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:41.624913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39821
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:41.705760 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:41.705962 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:41.705987 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:41.707158 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:41.707201 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:41.707226 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:41.708830 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:41.709016 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:41.709275 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:41.714545 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.714548 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.714567 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:41.714852 4165 server_base.cc:1034] running on GCE node
I20260928 09:04:41.715095 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:41.715337 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:41.716477 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790586281716452 us; error 36 us; skew 500 ppm
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:41.718959 4165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:41.719344 4165 webserver.cc:460] Webserver started at http://127.0.98.2:35801/ using document root <none> and password file <none>
I20260928 09:04:41.719524 4165 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:41.719563 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:41.719650 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:41.720623 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "680a4a2ca6734718872e91277604aef1"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "c2079878c6109d2977c1087fc6d160e9"
server_key_iv: "181461b0abe6ea509fd78946af3ec76e"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.720916 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "680a4a2ca6734718872e91277604aef1"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "c2079878c6109d2977c1087fc6d160e9"
server_key_iv: "181461b0abe6ea509fd78946af3ec76e"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.722247 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:04:41.722998 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.723189 4165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:41.723292 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "680a4a2ca6734718872e91277604aef1"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "c2079878c6109d2977c1087fc6d160e9"
server_key_iv: "181461b0abe6ea509fd78946af3ec76e"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.723384 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:41.735487 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:41.737331 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:41.737504 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:41.737758 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:41.738101 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:41.738162 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.738204 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:41.738226 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.743963 4165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41521
I20260928 09:04:41.744040 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41521 every 8 connection(s)
I20260928 09:04:41.744376 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:41.749807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4165
I20260928 09:04:41.749924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:41.750110 392 external_mini_cluster.cc:1428] Setting key e82db252ec3ab7035deb2255ecfb4ac3
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:41.754781 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.745144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51573 (local address 127.0.98.62:39821)
0928 09:04:41.745195 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:41.745197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:41.745407 (+ 210us) server_negotiation.cc:407] Connection header received
0928 09:04:41.746430 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:41.746431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:41.746441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:41.746502 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:41.747132 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.747621 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.748662 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.748849 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.748905 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.751437 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:41.751439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:41.751439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:41.751447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:41.753147 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.753644 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.753645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.753646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.753703 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.753987 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.753988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.753989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.754106 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:41.754186 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:41.754532 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 09:04:41.754667 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:41.755162 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.745100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39821 (local address 127.0.98.2:51573)
0928 09:04:41.745284 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:41.745326 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:41.746319 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:41.746585 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:41.746588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:41.746855 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:41.747051 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:41.747056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.747709 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.747709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:41.748560 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:41.748563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.748926 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.748927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:41.749021 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.749632 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:41.749643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:41.751343 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:41.753241 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:41.753249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:41.753252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:41.753550 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:41.753791 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:41.753791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:41.753792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:41.753909 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:41.754177 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:41.754183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:41.754342 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 09:04:41.754475 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 09:04:41.754646 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260928 09:04:41.755854 4294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39821
I20260928 09:04:41.756012 4294 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:41.756208 4294 heartbeater.cc:511] Master 127.0.98.62:39821 requested a full tablet report, sending...
I20260928 09:04:41.756628 3969 ts_manager.cc:194] Registered new tserver with Master: 680a4a2ca6734718872e91277604aef1 (127.0.98.2:41521)
I20260928 09:04:41.757174 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51573
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:41.776676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39821
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:41.857021 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:41.857286 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:41.857326 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:41.858510 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:41.858587 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:41.858633 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:41.860073 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:41.860338 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:41.860581 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:41.865613 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.865617 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:41.865628 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:41.866021 4299 server_base.cc:1034] running on GCE node
I20260928 09:04:41.866199 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:41.866394 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:41.867573 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790586281867558 us; error 35 us; skew 500 ppm
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:41.870097 4299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:41.870579 4299 webserver.cc:460] Webserver started at http://127.0.98.3:38487/ using document root <none> and password file <none>
I20260928 09:04:41.870805 4299 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:41.870877 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:41.871038 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:41.872071 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "69570fb9153e4f15b65507a967d3d48a"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "f2698410bebe26815919613faade1980"
server_key_iv: "7e6519e171c6580f7e100309003e92b4"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.872483 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69570fb9153e4f15b65507a967d3d48a"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "f2698410bebe26815919613faade1980"
server_key_iv: "7e6519e171c6580f7e100309003e92b4"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.873842 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:41.874601 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.874763 4299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:41.874867 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "69570fb9153e4f15b65507a967d3d48a"
format_stamp: "Formatted at 2026-09-28 09:04:41 on dist-test-slave-smns"
server_key: "f2698410bebe26815919613faade1980"
server_key_iv: "7e6519e171c6580f7e100309003e92b4"
server_key_version: "encryptionkey@0"
I20260928 09:04:41.874953 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:41.895418 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:41.897387 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:41.897630 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:41.897891 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:41.898250 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:41.898313 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.898348 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:41.898378 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:41.904232 4299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36785
I20260928 09:04:41.904276 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36785 every 8 connection(s)
I20260928 09:04:41.904593 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:41.911506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4299
I20260928 09:04:41.911590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:41.911810 392 external_mini_cluster.cc:1428] Setting key d843ae3a94940cab73334b1580f433aa
I20260928 09:04:41.914481 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.905288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58981 (local address 127.0.98.62:39821)
0928 09:04:41.905355 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:41.905358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:41.905612 (+ 254us) server_negotiation.cc:407] Connection header received
0928 09:04:41.906643 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:41.906644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:41.906656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:41.906706 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:41.907334 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.907839 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.908979 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.909180 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.909263 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.911731 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:41.911733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:41.911734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:41.911743 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:41.913231 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.913677 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.913678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.913678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.913731 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.913955 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.913956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.913956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.914062 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:41.914111 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:41.914346 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 09:04:41.914407 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:04:41.914652 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.905318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39821 (local address 127.0.98.3:58981)
0928 09:04:41.905489 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:41.905526 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:41.906538 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:41.906799 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:41.906802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:41.907062 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:41.907253 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:41.907257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.907927 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.907928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:41.908901 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:41.908905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.909285 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.909286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:41.909464 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.910026 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:41.910034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:41.911601 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:41.913307 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:41.913313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:41.913315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:41.913609 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:41.913791 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:41.913791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:41.913792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:41.913888 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:41.914117 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:41.914121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:41.914232 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 09:04:41.914308 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:04:41.914445 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260928 09:04:41.915133 4428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39821
I20260928 09:04:41.915212 4428 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:41.915411 4428 heartbeater.cc:511] Master 127.0.98.62:39821 requested a full tablet report, sending...
I20260928 09:04:41.915769 3969 ts_manager.cc:194] Registered new tserver with Master: 69570fb9153e4f15b65507a967d3d48a (127.0.98.3:36785)
I20260928 09:04:41.916121 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58981
I20260928 09:04:41.923657 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:41.947441 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.939341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.0.98.1:45957)
0928 09:04:41.939510 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:41.939517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:41.939530 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:04:41.939566 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:41.939568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:41.939583 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:41.939651 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:41.940197 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.940904 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.941702 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.942039 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.942097 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.944293 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:41.944305 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:41.944305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:41.944316 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:41.946069 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.946514 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.946518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.946518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.946583 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.946815 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.946816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.946816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.946976 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:41.947164 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:41.947221 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 09:04:41.947305 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
Sep 28 09:04:41 dist-test-slave-smns krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:41.955186 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:41.948866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.98.62:39821)
0928 09:04:41.948930 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:41.948932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:41.948941 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:41.948970 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:41.948971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:41.948980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:41.949027 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:41.949484 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.950034 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.950687 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:41.950861 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:41.950910 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:41.952967 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:41.952968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:41.952969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:41.952975 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:41.954233 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.954545 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.954546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.954546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.954595 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:41.954759 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:41.954760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:41.954760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:41.954896 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:41.954976 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:41.955065 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:04:41.955123 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 09:04:41.955433 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48514
I20260928 09:04:41.955832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4031
I20260928 09:04:41.960309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4165
I20260928 09:04:41.964684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4299
I20260928 09:04:41.969103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 3937
2026-09-28T09:04:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1878 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:42 dist-test-slave-smns krb5kdc[4440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:42 dist-test-slave-smns krb5kdc[4440](info): set up 2 sockets
Sep 28 09:04:42 dist-test-slave-smns krb5kdc[4440](info): commencing operation
krb5kdc: starting...
W20260928 09:04:43.030229 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:43.156699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:43.239856 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:43.240077 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:43.240099 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:43.241214 4453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:43.241252 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:43.241267 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:43.241281 4453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:43.241292 4453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:43.242909 4453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:43.243114 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:43.243290 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:43.248301 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:43.248306 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:43.248526 4453 server_base.cc:1034] running on GCE node
W20260928 09:04:43.248368 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:43.248831 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:43.249150 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:43.250298 4453 hybrid_clock.cc:648] HybridClock initialized: now 1790586283250265 us; error 61 us; skew 500 ppm
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:43.253199 4453 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:43.253728 4453 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:43.253783 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:43.253947 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:43.255087 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0d27efc2fb71430a9c9be153be1e6787"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "ae76f1db805d4bdbbbf8cf9e56c39b78"
server_key_iv: "e8f21bd94ae20d47471e7b735eafd6af"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.255486 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d27efc2fb71430a9c9be153be1e6787"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "ae76f1db805d4bdbbbf8cf9e56c39b78"
server_key_iv: "e8f21bd94ae20d47471e7b735eafd6af"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.256829 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:43.257668 4467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.257881 4453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:43.257973 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0d27efc2fb71430a9c9be153be1e6787"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "ae76f1db805d4bdbbbf8cf9e56c39b78"
server_key_iv: "e8f21bd94ae20d47471e7b735eafd6af"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.258025 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:43.278191 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:43.280134 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:43.280256 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:43.284173 4453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41081
I20260928 09:04:43.284199 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41081 every 8 connection(s)
I20260928 09:04:43.284512 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:43.285521 4520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:43.287940 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787: Bootstrap starting.
I20260928 09:04:43.288568 4520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:43.288870 4520 log.cc:824] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:43.289651 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787: No bootstrap required, opened a new log
I20260928 09:04:43.291290 4520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d27efc2fb71430a9c9be153be1e6787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41081 } }
I20260928 09:04:43.291460 4520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:43.291499 4520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d27efc2fb71430a9c9be153be1e6787, State: Initialized, Role: FOLLOWER
I20260928 09:04:43.291702 4520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d27efc2fb71430a9c9be153be1e6787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41081 } }
I20260928 09:04:43.291790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4453
I20260928 09:04:43.291805 4520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:43.291849 4520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:43.291898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:43.291901 4520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:43.292085 392 external_mini_cluster.cc:1428] Setting key 845cdbf1aa7761f191d2e5b47ce9b152
I20260928 09:04:43.292997 4520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d27efc2fb71430a9c9be153be1e6787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41081 } }
I20260928 09:04:43.293128 4520 leader_election.cc:302] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d27efc2fb71430a9c9be153be1e6787; no voters:
I20260928 09:04:43.293339 4520 leader_election.cc:288] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:43.293371 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:43.293485 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [term 1 LEADER]: Becoming Leader. State: Replica: 0d27efc2fb71430a9c9be153be1e6787, State: Running, Role: LEADER
I20260928 09:04:43.293586 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d27efc2fb71430a9c9be153be1e6787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41081 } }
I20260928 09:04:43.293629 4520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:43.293953 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d27efc2fb71430a9c9be153be1e6787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d27efc2fb71430a9c9be153be1e6787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41081 } } }
I20260928 09:04:43.294057 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:43.294217 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d27efc2fb71430a9c9be153be1e6787. Latest consensus state: current_term: 1 leader_uuid: "0d27efc2fb71430a9c9be153be1e6787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d27efc2fb71430a9c9be153be1e6787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41081 } } }
I20260928 09:04:43.294304 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:43.294277 4529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:43.294953 4529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:43.296685 4529 catalog_manager.cc:1347] Generated new cluster ID: 8a0a48a8126b4d8984b210ccd6faae2d
I20260928 09:04:43.296737 4529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:43.303692 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.292619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.98.62:41081)
0928 09:04:43.292806 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.292814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.292840 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:04:43.292941 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.292946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.292988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.293142 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:43.293945 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.295312 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.296477 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.297066 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.297167 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.299891 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:43.299895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:43.299895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:43.299907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:43.302020 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.302429 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.302432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.302433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.302553 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.302770 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.302771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.302771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.303096 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:43.303263 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.303333 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.303488 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 09:04:43.310979 4529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:43.311554 4529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:43.319900 4529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787: Generated new TSK 0
I20260928 09:04:43.320124 4529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:43.352960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41081
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:43.435300 4546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:43.435510 4546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:43.435534 4546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:43.436743 4546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:43.436789 4546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:43.436816 4546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:43.438414 4546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:43.438678 4546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:43.438930 4546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:43.444001 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:43.444145 4546 server_base.cc:1034] running on GCE node
W20260928 09:04:43.444293 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:43.444303 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:43.444586 4546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:43.444816 4546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:43.445996 4546 hybrid_clock.cc:648] HybridClock initialized: now 1790586283445986 us; error 54 us; skew 500 ppm
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:43.448683 4546 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:43.449074 4546 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:43.449123 4546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:43.449235 4546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:43.450328 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d490d55bd5540468b85ca68fd39be3a"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "f06413b1f291ffd19956cf46798fd1d2"
server_key_iv: "4283783988666113ed3712aa058009a3"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.450649 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d490d55bd5540468b85ca68fd39be3a"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "f06413b1f291ffd19956cf46798fd1d2"
server_key_iv: "4283783988666113ed3712aa058009a3"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.452008 4546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:43.452821 4560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.453122 4546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:43.453209 4546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "9d490d55bd5540468b85ca68fd39be3a"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "f06413b1f291ffd19956cf46798fd1d2"
server_key_iv: "4283783988666113ed3712aa058009a3"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.453270 4546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:43.482604 4546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:43.485113 4546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:43.485234 4546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:43.485433 4546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:43.485821 4546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:43.485865 4546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.485889 4546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:43.485931 4546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.492177 4546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42973
I20260928 09:04:43.492272 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42973 every 8 connection(s)
I20260928 09:04:43.492568 4546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:43.498598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4546
I20260928 09:04:43.498692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:43.498931 392 external_mini_cluster.cc:1428] Setting key da4e399bd8bbd5fbb37ce56c53a5fbf8
I20260928 09:04:43.502620 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.493344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34031 (local address 127.0.98.62:41081)
0928 09:04:43.493392 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.493394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.493658 (+ 264us) server_negotiation.cc:407] Connection header received
0928 09:04:43.494175 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.494176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.494190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.494253 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:43.495398 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.495928 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.496934 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.497134 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.497197 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.499473 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:43.499474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:43.499474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:43.499481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:43.501061 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.501514 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.501515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.501516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.501626 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.501953 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.501954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.501954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.502064 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:43.502150 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.502444 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.502527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:04:43.502817 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.493291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41081 (local address 127.0.98.1:34031)
0928 09:04:43.493457 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.493521 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.494079 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.494308 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.494311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.494704 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:43.495297 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.495302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.496025 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.496027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.496831 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.496833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.497217 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.497218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.497316 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.497847 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:43.497856 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:43.499381 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:43.501161 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:43.501167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:43.501170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:43.501413 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:43.501707 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:43.501708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:43.501708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:43.501827 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:43.502141 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:43.502148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:43.502299 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.502418 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.502584 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 09:04:43.503293 4674 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41081
I20260928 09:04:43.503410 4674 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:43.503592 4674 heartbeater.cc:511] Master 127.0.98.62:41081 requested a full tablet report, sending...
I20260928 09:04:43.504143 4484 ts_manager.cc:194] Registered new tserver with Master: 9d490d55bd5540468b85ca68fd39be3a (127.0.98.1:42973)
I20260928 09:04:43.504748 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34031
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:43.525236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41081
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:43.607717 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:43.607990 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:43.608037 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:43.609267 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:43.609347 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:43.609395 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:43.611147 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:43.611439 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:43.611726 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:43.617026 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:43.617113 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:43.617022 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:43.617358 4679 server_base.cc:1034] running on GCE node
I20260928 09:04:43.617646 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:43.617954 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:43.619145 4679 hybrid_clock.cc:648] HybridClock initialized: now 1790586283619133 us; error 26 us; skew 500 ppm
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:43.621845 4679 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:43.622220 4679 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:43.622273 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:43.622465 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:43.623667 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "6922061dacc348b992a5d93cfb36ba07"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "68fcb1165217a2c8f08a17cc689374a3"
server_key_iv: "fb348d7d379a4e98b7f5a74c90c5d7c9"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.624056 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6922061dacc348b992a5d93cfb36ba07"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "68fcb1165217a2c8f08a17cc689374a3"
server_key_iv: "fb348d7d379a4e98b7f5a74c90c5d7c9"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.625483 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:43.626300 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.626497 4679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:43.626619 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "6922061dacc348b992a5d93cfb36ba07"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "68fcb1165217a2c8f08a17cc689374a3"
server_key_iv: "fb348d7d379a4e98b7f5a74c90c5d7c9"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.626703 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:43.639779 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:43.641820 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:43.641990 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:43.642230 4679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:43.642566 4679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:43.642627 4679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.642669 4679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:43.642692 4679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.648785 4679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44785
I20260928 09:04:43.648818 4806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44785 every 8 connection(s)
I20260928 09:04:43.649097 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:43.650219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4679
I20260928 09:04:43.650348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:43.650565 392 external_mini_cluster.cc:1428] Setting key 42d69b3c783d88e2daa03de642b95e89
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:04:43.660104 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.649836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58153 (local address 127.0.98.62:41081)
0928 09:04:43.649927 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.649930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.650312 (+ 382us) server_negotiation.cc:407] Connection header received
0928 09:04:43.651039 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.651040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.651053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.651097 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:43.652135 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.653015 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.654161 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.654503 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.654588 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.656926 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:43.656928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:43.656929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:43.656939 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:43.658609 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.659077 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.659078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.659079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.659141 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.659418 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.659419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.659419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.659545 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:43.659627 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.659912 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.660006 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 09:04:43.660313 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.649806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41081 (local address 127.0.98.2:58153)
0928 09:04:43.650082 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.650121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.650933 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.651171 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.651176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.651512 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:43.652022 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.652028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.653136 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.653137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.654066 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.654068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.654606 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.654607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.654731 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.655323 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:43.655331 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:43.656815 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:43.658701 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:43.658709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:43.658711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:43.658947 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:43.659213 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:43.659214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:43.659215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:43.659340 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:43.659617 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:43.659623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:43.659773 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.659893 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.660072 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
I20260928 09:04:43.660856 4807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41081
I20260928 09:04:43.660941 4807 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:43.661137 4807 heartbeater.cc:511] Master 127.0.98.62:41081 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 09:04:43.661593 4484 ts_manager.cc:194] Registered new tserver with Master: 6922061dacc348b992a5d93cfb36ba07 (127.0.98.2:44785)
I20260928 09:04:43.662086 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:43.677314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41081
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:43.759878 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:43.760129 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:43.760170 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:43.761380 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:43.761461 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:43.761511 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:43.763892 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:43.764213 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:43.764451 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:43.769963 4820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:43.770037 4817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:43.769969 4818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:43.770251 4812 server_base.cc:1034] running on GCE node
I20260928 09:04:43.770469 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:43.770771 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:43.771972 4812 hybrid_clock.cc:648] HybridClock initialized: now 1790586283771948 us; error 38 us; skew 500 ppm
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:43.774614 4812 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:43.775045 4812 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:43.775123 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:43.775285 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:43.776355 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "10de1c421fab4c12a89ee31e1322eb6d"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "1722d482629f6d3b8bb5eb80fea8d00d"
server_key_iv: "d306a47c603bdac21ae63ac9ef9ff71d"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.776722 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "10de1c421fab4c12a89ee31e1322eb6d"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "1722d482629f6d3b8bb5eb80fea8d00d"
server_key_iv: "d306a47c603bdac21ae63ac9ef9ff71d"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.778129 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:04:43.778908 4826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.779088 4812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:43.779191 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "10de1c421fab4c12a89ee31e1322eb6d"
format_stamp: "Formatted at 2026-09-28 09:04:43 on dist-test-slave-smns"
server_key: "1722d482629f6d3b8bb5eb80fea8d00d"
server_key_iv: "d306a47c603bdac21ae63ac9ef9ff71d"
server_key_version: "encryptionkey@0"
I20260928 09:04:43.779273 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:43.801126 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:43.803288 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:43.803464 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:43.803714 4812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:43.804050 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:43.804109 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.804149 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:43.804170 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:43.810575 4812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36919
I20260928 09:04:43.810607 4939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36919 every 8 connection(s)
I20260928 09:04:43.810952 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:43.812297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4812
I20260928 09:04:43.812387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:43.812600 392 external_mini_cluster.cc:1428] Setting key 3d08fea848b54711a19fc1aad482fa27
Sep 28 09:04:43 dist-test-slave-smns krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:43.821314 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.811891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56713 (local address 127.0.98.62:41081)
0928 09:04:43.811959 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.811961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.812228 (+ 267us) server_negotiation.cc:407] Connection header received
0928 09:04:43.813031 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.813032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.813046 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.813105 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:43.814372 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.814883 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.815888 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.816094 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.816165 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.818405 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:43.818407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:43.818407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:43.818414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:43.819657 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.820259 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.820260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.820261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.820328 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.820601 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.820602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.820603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.820728 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:43.820813 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.821137 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.821222 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:04:43.821501 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.811881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41081 (local address 127.0.98.3:56713)
0928 09:04:43.812055 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.812092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.812856 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.813203 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.813207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.813677 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:43.814257 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.814264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.814986 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.814986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.815782 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.815784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.816183 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.816183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.816286 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.816857 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:43.816874 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:43.818307 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:43.819777 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:43.819784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:43.819786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:43.820164 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:43.820409 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:43.820409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:43.820410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:43.820530 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:43.820803 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:43.820809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:43.820992 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.821104 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.821266 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260928 09:04:43.821955 4940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41081
I20260928 09:04:43.822036 4940 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:43.822230 4940 heartbeater.cc:511] Master 127.0.98.62:41081 requested a full tablet report, sending...
I20260928 09:04:43.822670 4484 ts_manager.cc:194] Registered new tserver with Master: 10de1c421fab4c12a89ee31e1322eb6d (127.0.98.3:36919)
I20260928 09:04:43.823062 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56713
I20260928 09:04:43.824501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:04:43.832136 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.825920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.0.98.62:41081)
0928 09:04:43.825995 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.825997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.826093 (+ 96us) server_negotiation.cc:407] Connection header received
0928 09:04:43.826181 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.826183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.826194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.826265 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:43.826841 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.827315 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.828080 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.828266 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.828343 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.829572 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:43.829573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:43.829574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:43.829581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:43.830957 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.831321 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.831322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.831322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.831380 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:43.831595 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:43.831596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:43.831596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:43.831739 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:43.831848 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.831958 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.832036 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:04:43.833961 4484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:43.835582 4484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:43.841578 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.837749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42973 (local address 127.0.0.1:57724)
0928 09:04:43.837897 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.837939 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.838020 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.838247 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.838251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.838253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:43.838322 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.838323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.839372 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.839373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.840350 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.840352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.841257 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.841258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.841342 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.841358 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.841432 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.841482 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":46,"threads_started":1}
I20260928 09:04:43.841943 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.837881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.1:42973)
0928 09:04:43.838059 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.838065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.838077 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:43.838115 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.838117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.838133 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.838218 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:43.838391 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.839266 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.840448 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.841172 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.841256 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.841317 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.841456 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.841492 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260928 09:04:43.842226 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.837750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36919 (local address 127.0.0.1:45062)
0928 09:04:43.838391 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.838401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.838438 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.838725 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.838726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.838728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:43.838781 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.838783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.840068 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.840069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.841080 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.841083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.841992 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.841993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.842053 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.842058 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.842116 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.842155 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":29,"thread_start_us":94,"threads_started":1}
I20260928 09:04:43.842239 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.837987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.0.98.3:36919)
0928 09:04:43.838586 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.838590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.838600 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:43.838635 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.838637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.838653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.838728 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:43.838859 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.839939 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.841175 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.841882 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.841989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.842043 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.842130 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.842166 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":84,"threads_started":1}
I20260928 09:04:43.842694 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.838052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.0.98.2:44785)
0928 09:04:43.838196 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.838200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.838289 (+ 89us) server_negotiation.cc:407] Connection header received
0928 09:04:43.838544 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.838547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.838563 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.838639 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:43.838831 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.839950 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.841622 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.842366 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.842439 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.842498 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.842583 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.842618 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260928 09:04:43.842751 4606 tablet_service.cc:1467] Processing CreateTablet for tablet d32462ae758b48218f40e0c26648618c (DEFAULT_TABLE table=test-table [id=c63d9f0f8a2a43249d1b2a5d181c5912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:43.842955 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.837750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44785 (local address 127.0.0.1:54222)
0928 09:04:43.838203 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.838214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.838276 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.838723 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.838724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.838726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:43.838768 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.838769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.840072 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.840073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.841212 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.841215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.842438 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.842439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.842505 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.842508 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.842570 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.842881 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":25,"thread_start_us":163,"threads_started":1}
I20260928 09:04:43.842896 4870 tablet_service.cc:1467] Processing CreateTablet for tablet d32462ae758b48218f40e0c26648618c (DEFAULT_TABLE table=test-table [id=c63d9f0f8a2a43249d1b2a5d181c5912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:43.843094 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32462ae758b48218f40e0c26648618c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:43.843214 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32462ae758b48218f40e0c26648618c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:43.843634 4741 tablet_service.cc:1467] Processing CreateTablet for tablet d32462ae758b48218f40e0c26648618c (DEFAULT_TABLE table=test-table [id=c63d9f0f8a2a43249d1b2a5d181c5912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:43.843923 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32462ae758b48218f40e0c26648618c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:43.845671 4957 tablet_bootstrap.cc:492] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: Bootstrap starting.
I20260928 09:04:43.846354 4957 tablet_bootstrap.cc:654] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:43.846740 4958 tablet_bootstrap.cc:492] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Bootstrap starting.
I20260928 09:04:43.846959 4957 log.cc:824] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:43.847214 4959 tablet_bootstrap.cc:492] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Bootstrap starting.
I20260928 09:04:43.847415 4958 tablet_bootstrap.cc:654] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:43.847808 4959 tablet_bootstrap.cc:654] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:43.847990 4958 log.cc:824] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:43.848431 4959 log.cc:824] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:43.848788 4958 tablet_bootstrap.cc:492] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: No bootstrap required, opened a new log
I20260928 09:04:43.848953 4958 ts_tablet_manager.cc:1397] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:04:43.849136 4957 tablet_bootstrap.cc:492] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: No bootstrap required, opened a new log
I20260928 09:04:43.849135 4959 tablet_bootstrap.cc:492] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: No bootstrap required, opened a new log
I20260928 09:04:43.849205 4959 ts_tablet_manager.cc:1397] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:04:43.849205 4957 ts_tablet_manager.cc:1397] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 09:04:43.850483 4958 raft_consensus.cc:348] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.850641 4958 raft_consensus.cc:374] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:43.850683 4958 raft_consensus.cc:729] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10de1c421fab4c12a89ee31e1322eb6d, State: Initialized, Role: FOLLOWER
I20260928 09:04:43.850821 4958 consensus_queue.cc:260] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.851126 4958 ts_tablet_manager.cc:1428] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:43.851257 4940 heartbeater.cc:503] Master 127.0.98.62:41081 was elected leader, sending a full tablet report...
I20260928 09:04:43.851240 4957 raft_consensus.cc:348] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.851238 4959 raft_consensus.cc:348] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.851397 4957 raft_consensus.cc:374] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:43.851397 4959 raft_consensus.cc:374] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:43.851434 4959 raft_consensus.cc:729] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6922061dacc348b992a5d93cfb36ba07, State: Initialized, Role: FOLLOWER
I20260928 09:04:43.851434 4957 raft_consensus.cc:729] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d490d55bd5540468b85ca68fd39be3a, State: Initialized, Role: FOLLOWER
I20260928 09:04:43.851557 4957 consensus_queue.cc:260] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.851558 4959 consensus_queue.cc:260] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.851825 4959 ts_tablet_manager.cc:1428] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:04:43.851825 4957 ts_tablet_manager.cc:1428] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 09:04:43.851873 4674 heartbeater.cc:503] Master 127.0.98.62:41081 was elected leader, sending a full tablet report...
I20260928 09:04:43.851933 4807 heartbeater.cc:503] Master 127.0.98.62:41081 was elected leader, sending a full tablet report...
I20260928 09:04:43.893920 4964 raft_consensus.cc:482] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:43.894088 4964 raft_consensus.cc:504] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.894538 4964 leader_election.cc:288] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10de1c421fab4c12a89ee31e1322eb6d (127.0.98.3:36919), 9d490d55bd5540468b85ca68fd39be3a (127.0.98.1:42973)
I20260928 09:04:43.897684 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.894829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33623 (local address 127.0.98.3:36919)
0928 09:04:43.894904 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.894907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.894918 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:43.894949 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.894950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.894962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.895022 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:43.895212 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.895796 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.896799 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.897311 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.897365 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.897374 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.897569 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.897607 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:04:43.897673 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.894685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36919 (local address 127.0.98.2:33623)
0928 09:04:43.894777 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.894790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.894841 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.895046 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.895047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.895049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:43.895126 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.895128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.895877 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.895878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.896693 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.896695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.897394 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.897395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.897476 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.897481 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.897536 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.897602 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":29}
I20260928 09:04:43.897841 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.894724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42973 (local address 127.0.98.2:50813)
0928 09:04:43.894930 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.894941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:43.895005 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:43.895126 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:43.895127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:43.895128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:43.895225 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.895227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.895946 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.895947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.896757 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:43.896759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.897578 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.897579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:43.897645 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.897649 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:43.897709 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 09:04:43.897751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260928 09:04:43.897912 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.894818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50813 (local address 127.0.98.1:42973)
0928 09:04:43.894920 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.894922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.895027 (+ 105us) server_negotiation.cc:407] Connection header received
0928 09:04:43.895054 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.895055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.895067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.895125 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:43.895303 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.895823 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.896908 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.897435 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.897515 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.897527 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.897732 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.897780 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 09:04:43.898078 4894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32462ae758b48218f40e0c26648618c" candidate_uuid: "6922061dacc348b992a5d93cfb36ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10de1c421fab4c12a89ee31e1322eb6d" is_pre_election: true
I20260928 09:04:43.898212 4894 raft_consensus.cc:2454] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6922061dacc348b992a5d93cfb36ba07 in term 0.
I20260928 09:04:43.898443 4696 leader_election.cc:302] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10de1c421fab4c12a89ee31e1322eb6d, 6922061dacc348b992a5d93cfb36ba07; no voters:
I20260928 09:04:43.898579 4964 raft_consensus.cc:2781] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:43.898563 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32462ae758b48218f40e0c26648618c" candidate_uuid: "6922061dacc348b992a5d93cfb36ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d490d55bd5540468b85ca68fd39be3a" is_pre_election: true
I20260928 09:04:43.898633 4964 raft_consensus.cc:482] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:43.898656 4964 raft_consensus.cc:3037] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:43.898705 4628 raft_consensus.cc:2454] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6922061dacc348b992a5d93cfb36ba07 in term 0.
I20260928 09:04:43.899386 4964 raft_consensus.cc:504] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.899552 4964 leader_election.cc:288] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [CANDIDATE]: Term 1 election: Requested vote from peers 10de1c421fab4c12a89ee31e1322eb6d (127.0.98.3:36919), 9d490d55bd5540468b85ca68fd39be3a (127.0.98.1:42973)
W20260928 09:04:43.899633 4808 tablet.cc:2369] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:43.899655 4894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32462ae758b48218f40e0c26648618c" candidate_uuid: "6922061dacc348b992a5d93cfb36ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10de1c421fab4c12a89ee31e1322eb6d"
I20260928 09:04:43.899698 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32462ae758b48218f40e0c26648618c" candidate_uuid: "6922061dacc348b992a5d93cfb36ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d490d55bd5540468b85ca68fd39be3a"
I20260928 09:04:43.899721 4894 raft_consensus.cc:3037] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:43.899771 4628 raft_consensus.cc:3037] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:43.900389 4628 raft_consensus.cc:2454] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6922061dacc348b992a5d93cfb36ba07 in term 1.
I20260928 09:04:43.900542 4894 raft_consensus.cc:2454] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6922061dacc348b992a5d93cfb36ba07 in term 1.
I20260928 09:04:43.900563 4695 leader_election.cc:302] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6922061dacc348b992a5d93cfb36ba07, 9d490d55bd5540468b85ca68fd39be3a; no voters:
I20260928 09:04:43.900751 4964 raft_consensus.cc:2781] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:43.900933 4964 raft_consensus.cc:686] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 1 LEADER]: Becoming Leader. State: Replica: 6922061dacc348b992a5d93cfb36ba07, State: Running, Role: LEADER
I20260928 09:04:43.901018 4964 consensus_queue.cc:237] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } }
I20260928 09:04:43.901794 4483 catalog_manager.cc:5495] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6922061dacc348b992a5d93cfb36ba07 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6922061dacc348b992a5d93cfb36ba07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6922061dacc348b992a5d93cfb36ba07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44785 } health_report { overall_health: HEALTHY } } }
I20260928 09:04:43.916913 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.914345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.0.98.2:44785)
0928 09:04:43.914407 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:43.914410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:43.914417 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:43.914450 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:43.914450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:43.914470 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:43.914567 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:43.914740 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.915233 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.916070 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:43.916286 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:43.916368 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:43.916543 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:43.916654 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:43.916766 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:43.916789 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 09:04:43.916809 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 09:04:43.918701 4628 raft_consensus.cc:1261] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 1 FOLLOWER]: Refusing update from remote peer 6922061dacc348b992a5d93cfb36ba07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:43.918720 4894 raft_consensus.cc:1261] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 1 FOLLOWER]: Refusing update from remote peer 6922061dacc348b992a5d93cfb36ba07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:43.919085 4967 consensus_queue.cc:1035] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d490d55bd5540468b85ca68fd39be3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:43.919210 4964 consensus_queue.cc:1035] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10de1c421fab4c12a89ee31e1322eb6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:43.922142 4971 mvcc.cc:205] Tried to move back new op lower bound from 7334241418928410624 to 7334241418859094016. Current Snapshot: MvccSnapshot[applied={T|T < 7334241418928410624}]
I20260928 09:04:43.924094 4483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:43.924211 4483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:43.924934 4483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d27efc2fb71430a9c9be153be1e6787: Sending DeleteTablet for 3 replicas of tablet d32462ae758b48218f40e0c26648618c
I20260928 09:04:43.925309 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet d32462ae758b48218f40e0c26648618c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57724
I20260928 09:04:43.925308 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet d32462ae758b48218f40e0c26648618c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54222
I20260928 09:04:43.925334 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet d32462ae758b48218f40e0c26648618c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45062
I20260928 09:04:43.925594 4982 tablet_replica.cc:330] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a: stopping tablet replica
I20260928 09:04:43.925627 4981 tablet_replica.cc:330] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: stopping tablet replica
I20260928 09:04:43.925694 4982 raft_consensus.cc:2229] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:43.925786 4982 raft_consensus.cc:2258] T d32462ae758b48218f40e0c26648618c P 9d490d55bd5540468b85ca68fd39be3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:43.925904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4546
I20260928 09:04:43.925905 4981 raft_consensus.cc:2229] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:43.927385 4981 raft_consensus.cc:2258] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:43.928016 4981 ts_tablet_manager.cc:1905] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:43.930462 4981 ts_tablet_manager.cc:1918] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:43.930598 4981 log.cc:1196] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/d32462ae758b48218f40e0c26648618c
I20260928 09:04:43.930956 4981 ts_tablet_manager.cc:1939] T d32462ae758b48218f40e0c26648618c P 6922061dacc348b992a5d93cfb36ba07: Deleting consensus metadata
I20260928 09:04:43.931447 4468 catalog_manager.cc:4841] TS 6922061dacc348b992a5d93cfb36ba07 (127.0.98.2:44785): tablet d32462ae758b48218f40e0c26648618c (table test-table [id=c63d9f0f8a2a43249d1b2a5d181c5912]) successfully deleted
W20260928 09:04:43.932543 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42973 (error 108)
I20260928 09:04:43.932865 4980 tablet_replica.cc:330] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: stopping tablet replica
I20260928 09:04:43.932998 4980 raft_consensus.cc:2229] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:43.933081 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:43.932940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42973 (local address 127.0.0.1:57738)
0928 09:04:43.933005 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:43.933048 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 09:04:43.933121 4980 raft_consensus.cc:2258] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 09:04:43.933254 4469 catalog_manager.cc:4584] TS 9d490d55bd5540468b85ca68fd39be3a (127.0.98.1:42973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d32462ae758b48218f40e0c26648618c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42973: connect: Connection refused (error 111)
I20260928 09:04:43.933316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4679
I20260928 09:04:43.935081 4980 ts_tablet_manager.cc:1905] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:43.936729 4980 ts_tablet_manager.cc:1918] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:43.936794 4980 log.cc:1196] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/d32462ae758b48218f40e0c26648618c
I20260928 09:04:43.937085 4980 ts_tablet_manager.cc:1939] T d32462ae758b48218f40e0c26648618c P 10de1c421fab4c12a89ee31e1322eb6d: Deleting consensus metadata
I20260928 09:04:43.937497 4470 catalog_manager.cc:4841] TS 10de1c421fab4c12a89ee31e1322eb6d (127.0.98.3:36919): tablet d32462ae758b48218f40e0c26648618c (table test-table [id=c63d9f0f8a2a43249d1b2a5d181c5912]) successfully deleted
I20260928 09:04:43.938761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4812
I20260928 09:04:43.943409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4453
2026-09-28T09:04:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1973 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T09:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:43Z Disabled control of system clock
I20260928 09:04:43.970305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43879
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 09:04:44.051528 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:44.051759 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:44.051781 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:44.052924 4987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:44.052965 4987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:44.052978 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:44.052990 4987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:44.053002 4987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:44.054425 4987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:44.054634 4987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:44.054865 4987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:44.059988 4995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:44.060015 4993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.060197 4987 server_base.cc:1034] running on GCE node
W20260928 09:04:44.060026 4992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.060518 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:44.060846 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:44.062014 4987 hybrid_clock.cc:648] HybridClock initialized: now 1790586284061987 us; error 37 us; skew 500 ppm
I20260928 09:04:44.063300 4987 webserver.cc:460] Webserver started at http://127.0.98.62:35651/ using document root <none> and password file <none>
I20260928 09:04:44.063567 4987 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:44.063642 4987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:44.063797 4987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:44.064888 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "b6983868c5e946a7aa1227c0fefb1b41"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "3062866b0886b25c09928c5dcd25a2c1"
server_key_iv: "e9588632ed34a806ad4ee0bfe447b49c"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.065267 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6983868c5e946a7aa1227c0fefb1b41"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "3062866b0886b25c09928c5dcd25a2c1"
server_key_iv: "e9588632ed34a806ad4ee0bfe447b49c"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.066656 4987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:44.067406 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.067584 4987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:44.067693 4987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "b6983868c5e946a7aa1227c0fefb1b41"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "3062866b0886b25c09928c5dcd25a2c1"
server_key_iv: "e9588632ed34a806ad4ee0bfe447b49c"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.067777 4987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:44.071200 4987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:44.071481 4987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:44.071622 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:44.075623 4987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43879
I20260928 09:04:44.075649 5053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43879 every 8 connection(s)
I20260928 09:04:44.075975 4987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:44.076660 5054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:44.079492 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41: Bootstrap starting.
I20260928 09:04:44.080009 5054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:44.080245 5054 log.cc:824] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:44.080864 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41: No bootstrap required, opened a new log
I20260928 09:04:44.082492 5054 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6983868c5e946a7aa1227c0fefb1b41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260928 09:04:44.082628 5054 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:44.082650 5054 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6983868c5e946a7aa1227c0fefb1b41, State: Initialized, Role: FOLLOWER
I20260928 09:04:44.082793 5054 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6983868c5e946a7aa1227c0fefb1b41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260928 09:04:44.082875 5054 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:44.082898 5054 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:44.082926 5054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:44.083395 5054 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6983868c5e946a7aa1227c0fefb1b41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260928 09:04:44.083485 5054 leader_election.cc:302] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6983868c5e946a7aa1227c0fefb1b41; no voters:
I20260928 09:04:44.083611 5054 leader_election.cc:288] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:44.083675 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:44.083854 5054 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:44.083873 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [term 1 LEADER]: Becoming Leader. State: Replica: b6983868c5e946a7aa1227c0fefb1b41, State: Running, Role: LEADER
I20260928 09:04:44.084072 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6983868c5e946a7aa1227c0fefb1b41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260928 09:04:44.084239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 4987
I20260928 09:04:44.084335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:44.084477 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6983868c5e946a7aa1227c0fefb1b41. Latest consensus state: current_term: 1 leader_uuid: "b6983868c5e946a7aa1227c0fefb1b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6983868c5e946a7aa1227c0fefb1b41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } } }
I20260928 09:04:44.084525 392 external_mini_cluster.cc:1428] Setting key 1a48ac4122ac987623b8a677e70f88eb
I20260928 09:04:44.084563 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:44.084695 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6983868c5e946a7aa1227c0fefb1b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6983868c5e946a7aa1227c0fefb1b41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } } }
I20260928 09:04:44.084749 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:44.085427 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:44.085952 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:44.087195 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.085137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.0.98.62:43879)
0928 09:04:44.085588 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.085599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.085635 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:04:44.085757 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.085764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.085814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.085959 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.086702 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.086705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.086706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.086710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.086774 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.086790 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.086831 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.086959 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.087016 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":62,"threads_started":1}
I20260928 09:04:44.087543 5066 catalog_manager.cc:1347] Generated new cluster ID: 188d21a357844b0f8745c9ad8f3503ee
I20260928 09:04:44.087594 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:44.126176 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:44.126384 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:44.131271 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41: Generated new TSK 0
I20260928 09:04:44.131467 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:04:44.140628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--builtin_ntp_servers=127.0.98.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 09:04:44.223042 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:44.223237 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:44.223259 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:44.224395 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:44.224433 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:44.224459 5078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:44.225967 5078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:44.226233 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:44.226545 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:44.231626 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:44.231691 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.231691 5078 server_base.cc:1034] running on GCE node
W20260928 09:04:44.231887 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.232151 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:44.232417 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:44.233623 5078 hybrid_clock.cc:648] HybridClock initialized: now 1790586284233608 us; error 30 us; skew 500 ppm
I20260928 09:04:44.235080 5078 webserver.cc:460] Webserver started at http://127.0.98.1:35085/ using document root <none> and password file <none>
I20260928 09:04:44.235320 5078 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:44.235404 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:44.235555 5078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:44.236579 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "c88494efbeba4cea82cd074290ef23ff"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "dff0846893f938bb88d78665644e6c24"
server_key_iv: "a59b747dc0b6fc895ed1138756663026"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.236953 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c88494efbeba4cea82cd074290ef23ff"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "dff0846893f938bb88d78665644e6c24"
server_key_iv: "a59b747dc0b6fc895ed1138756663026"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.238356 5078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:44.239140 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.239324 5078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:44.239428 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "c88494efbeba4cea82cd074290ef23ff"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "dff0846893f938bb88d78665644e6c24"
server_key_iv: "a59b747dc0b6fc895ed1138756663026"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.239512 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:44.242973 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:44.243248 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:44.243397 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:44.243603 5078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:44.243908 5078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:44.243968 5078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.244009 5078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:44.244036 5078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.249912 5078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37973
I20260928 09:04:44.249948 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37973 every 8 connection(s)
I20260928 09:04:44.250315 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:44.253645 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.251086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43879 (local address 127.0.98.1:39819)
0928 09:04:44.251290 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.251325 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.251776 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.252020 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.252023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.252031 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.252947 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.252947 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.252977 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.253006 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.253285 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.253289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.253292 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.253352 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.253403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 09:04:44.253710 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.251090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39819 (local address 127.0.98.62:43879)
0928 09:04:44.251156 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.251159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.251395 (+ 236us) server_negotiation.cc:407] Connection header received
0928 09:04:44.251875 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.251876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.251885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.251960 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.253129 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.253130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.253131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.253150 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.253185 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.253190 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.253282 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.253537 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.253625 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:04:44.254248 5206 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43879
I20260928 09:04:44.254340 5206 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:44.254498 5206 heartbeater.cc:511] Master 127.0.98.62:43879 requested a full tablet report, sending...
I20260928 09:04:44.255105 5018 ts_manager.cc:194] Registered new tserver with Master: c88494efbeba4cea82cd074290ef23ff (127.0.98.1:37973)
I20260928 09:04:44.255559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5078
I20260928 09:04:44.255685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:44.255921 392 external_mini_cluster.cc:1428] Setting key f5daae42b9d31291a2fdac4f4e64460e
I20260928 09:04:44.256891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--builtin_ntp_servers=127.0.98.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 09:04:44.338510 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:44.338713 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:44.338737 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:44.339962 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:44.340005 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:44.340031 5209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:44.341403 5209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:44.341686 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:44.342015 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:44.347275 5214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:44.347263 5215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.347384 5209 server_base.cc:1034] running on GCE node
W20260928 09:04:44.347266 5217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.347700 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:44.347929 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:44.349112 5209 hybrid_clock.cc:648] HybridClock initialized: now 1790586284349107 us; error 34 us; skew 500 ppm
I20260928 09:04:44.350369 5209 webserver.cc:460] Webserver started at http://127.0.98.2:36079/ using document root <none> and password file <none>
I20260928 09:04:44.350626 5209 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:44.350700 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:44.350847 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:44.351954 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ed25957c9554e2a8b270503793f88af"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "4c5868feabcb9de54d337c7a8d5b6419"
server_key_iv: "c1a3f1be073b7090f33d700a85e0d1af"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.352335 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ed25957c9554e2a8b270503793f88af"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "4c5868feabcb9de54d337c7a8d5b6419"
server_key_iv: "c1a3f1be073b7090f33d700a85e0d1af"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.353683 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:44.354554 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.354729 5209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:44.354830 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "1ed25957c9554e2a8b270503793f88af"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "4c5868feabcb9de54d337c7a8d5b6419"
server_key_iv: "c1a3f1be073b7090f33d700a85e0d1af"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.354960 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:44.358322 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:44.358595 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:44.358737 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:44.358944 5209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:44.359232 5209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:44.359288 5209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.359328 5209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:44.359357 5209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.365298 5209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42493
I20260928 09:04:44.365358 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42493 every 8 connection(s)
I20260928 09:04:44.365743 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:44.368688 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.366573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47817 (local address 127.0.98.62:43879)
0928 09:04:44.366619 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.366621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.366849 (+ 228us) server_negotiation.cc:407] Connection header received
0928 09:04:44.367345 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.367347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.367353 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.367422 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.368489 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.368491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.368491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.368494 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.368507 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.368512 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.368562 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.368624 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.368636 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:04:44.368986 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.366520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43879 (local address 127.0.98.2:47817)
0928 09:04:44.366698 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.366736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.367201 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.367472 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.367476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.367483 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.368346 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.368346 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.368374 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.368402 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.368573 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.368575 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.368578 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.368632 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.368679 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 09:04:44.369907 5337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43879
I20260928 09:04:44.370000 5337 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:44.370107 5337 heartbeater.cc:511] Master 127.0.98.62:43879 requested a full tablet report, sending...
I20260928 09:04:44.370509 5018 ts_manager.cc:194] Registered new tserver with Master: 1ed25957c9554e2a8b270503793f88af (127.0.98.2:42493)
I20260928 09:04:44.371536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5209
I20260928 09:04:44.371650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:44.371852 392 external_mini_cluster.cc:1428] Setting key 667242d481e1b7cf67195650a7714e33
I20260928 09:04:44.373070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--builtin_ntp_servers=127.0.98.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 09:04:44.454851 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:44.455047 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:44.455068 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:44.456261 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:44.456339 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:44.456386 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:44.457885 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:44.458148 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:44.458500 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:44.463397 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:44.463470 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:44.463503 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.463518 5340 server_base.cc:1034] running on GCE node
I20260928 09:04:44.463841 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:44.464074 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:44.465251 5340 hybrid_clock.cc:648] HybridClock initialized: now 1790586284465243 us; error 47 us; skew 500 ppm
I20260928 09:04:44.466670 5340 webserver.cc:460] Webserver started at http://127.0.98.3:33215/ using document root <none> and password file <none>
I20260928 09:04:44.466948 5340 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:44.466996 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:44.467128 5340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:44.468209 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "7a5fb7b9855b4dd4b39617bbb7261137"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "9ecf1e058315e8702968193f4c2f77c1"
server_key_iv: "f3d130091a9805ec46b7bc0bf56d88a9"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.468494 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7a5fb7b9855b4dd4b39617bbb7261137"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "9ecf1e058315e8702968193f4c2f77c1"
server_key_iv: "f3d130091a9805ec46b7bc0bf56d88a9"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.469754 5340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:44.470613 5354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.470834 5340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:44.470986 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "7a5fb7b9855b4dd4b39617bbb7261137"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "9ecf1e058315e8702968193f4c2f77c1"
server_key_iv: "f3d130091a9805ec46b7bc0bf56d88a9"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.471083 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:44.474331 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:44.474591 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:44.474731 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:44.474936 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:44.475232 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:44.475289 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.475328 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:44.475354 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.480821 5340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41623
I20260928 09:04:44.480899 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41623 every 8 connection(s)
I20260928 09:04:44.481211 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:44.484436 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.481936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46805 (local address 127.0.98.62:43879)
0928 09:04:44.481994 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.481996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.482232 (+ 236us) server_negotiation.cc:407] Connection header received
0928 09:04:44.482771 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.482773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.482780 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.482832 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.484171 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.484173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.484173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.484178 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.484193 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.484197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.484245 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.484349 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.484366 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:04:44.484606 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.481901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43879 (local address 127.0.98.3:46805)
0928 09:04:44.482113 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.482146 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.482647 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.482911 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.482914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.482922 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.484021 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.484021 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.484049 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.484077 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.484251 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.484256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.484259 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.484352 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.484412 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260928 09:04:44.485042 5468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43879
I20260928 09:04:44.485136 5468 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:44.485308 5468 heartbeater.cc:511] Master 127.0.98.62:43879 requested a full tablet report, sending...
I20260928 09:04:44.485670 5018 ts_manager.cc:194] Registered new tserver with Master: 7a5fb7b9855b4dd4b39617bbb7261137 (127.0.98.3:41623)
I20260928 09:04:44.487586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5340
I20260928 09:04:44.487721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:44.487906 392 external_mini_cluster.cc:1428] Setting key b4e5342fa93fc25a0342331566055deb
I20260928 09:04:44.488960 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:04:44.491595 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.490302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.0.98.62:43879)
0928 09:04:44.490380 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.490382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.490458 (+ 76us) server_negotiation.cc:407] Connection header received
0928 09:04:44.490542 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.490543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.490548 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.490620 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.491352 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.491354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.491354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.491359 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.491374 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.491382 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.491447 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.491505 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.491521 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 09:04:44.493117 5018 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:44.494928 5018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:44.498687 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.497040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:51884)
0928 09:04:44.497401 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.497410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.497443 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.497711 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.497712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.497714 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.498239 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.498239 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.498256 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.498290 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.498522 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.498528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.498531 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.498601 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.498614 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":106,"threads_started":1}
I20260928 09:04:44.498939 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.497288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.0.98.1:37973)
0928 09:04:44.497485 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.497489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.497497 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:44.497519 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.497520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.497527 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.497676 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.498379 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.498385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.498386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.498389 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.498427 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.498441 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.498514 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.498828 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.498846 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260928 09:04:44.499046 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.497259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.0.98.3:41623)
0928 09:04:44.498029 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.498033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.498040 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:44.498058 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.498060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.498065 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.498126 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.498607 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.498612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.498613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.498616 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.498643 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.498660 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.498718 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.498932 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.498950 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":102,"threads_started":1}
I20260928 09:04:44.499150 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.497036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42493 (local address 127.0.0.1:53524)
0928 09:04:44.497214 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.497261 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.497317 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.497589 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.497594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.497600 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.498510 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.498511 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.498518 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.498531 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.499034 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.499035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.499036 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.499067 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.499079 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260928 09:04:44.498867 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.497041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.0.1:58382)
0928 09:04:44.497530 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.497538 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.497585 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.498149 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.498150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.498153 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.498511 (+ 358us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.498511 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.498518 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.498531 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.498729 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.498730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.498731 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.498785 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.498799 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":7,"thread_start_us":89,"threads_started":1}
I20260928 09:04:44.499359 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.497222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.98.2:42493)
0928 09:04:44.497414 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.497421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.497431 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:44.497456 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.497458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.497468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.498619 (+ 1151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.498891 (+ 272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.498895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.498895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.498899 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.498928 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.498942 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.499242 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.499267 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.499281 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260928 09:04:44.499755 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd99e096f9e4f59b493a298ab4f2746 (DEFAULT_TABLE table=test-table [id=1e24906f1f904d59b2e9f7aeebafb376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:44.499861 5271 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd99e096f9e4f59b493a298ab4f2746 (DEFAULT_TABLE table=test-table [id=1e24906f1f904d59b2e9f7aeebafb376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:44.499912 5402 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd99e096f9e4f59b493a298ab4f2746 (DEFAULT_TABLE table=test-table [id=1e24906f1f904d59b2e9f7aeebafb376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:44.500113 5271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd99e096f9e4f59b493a298ab4f2746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:44.500113 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd99e096f9e4f59b493a298ab4f2746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:44.500270 5402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd99e096f9e4f59b493a298ab4f2746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:44.502839 5482 tablet_bootstrap.cc:492] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Bootstrap starting.
I20260928 09:04:44.503288 5484 tablet_bootstrap.cc:492] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Bootstrap starting.
I20260928 09:04:44.503623 5482 tablet_bootstrap.cc:654] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:44.503947 5482 log.cc:824] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:44.504022 5484 tablet_bootstrap.cc:654] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:44.504307 5484 log.cc:824] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:44.504630 5483 tablet_bootstrap.cc:492] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: Bootstrap starting.
I20260928 09:04:44.504688 5482 tablet_bootstrap.cc:492] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: No bootstrap required, opened a new log
I20260928 09:04:44.504746 5482 ts_tablet_manager.cc:1397] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:44.505021 5484 tablet_bootstrap.cc:492] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: No bootstrap required, opened a new log
I20260928 09:04:44.505105 5484 ts_tablet_manager.cc:1397] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:44.505272 5483 tablet_bootstrap.cc:654] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:44.505630 5483 log.cc:824] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:44.506982 5482 raft_consensus.cc:348] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.507110 5482 raft_consensus.cc:374] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:44.507133 5482 raft_consensus.cc:729] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5fb7b9855b4dd4b39617bbb7261137, State: Initialized, Role: FOLLOWER
I20260928 09:04:44.507261 5482 consensus_queue.cc:260] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.507196 5484 raft_consensus.cc:348] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.507368 5484 raft_consensus.cc:374] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:44.507416 5484 raft_consensus.cc:729] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed25957c9554e2a8b270503793f88af, State: Initialized, Role: FOLLOWER
I20260928 09:04:44.507526 5482 ts_tablet_manager.cc:1428] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:04:44.507562 5468 heartbeater.cc:503] Master 127.0.98.62:43879 was elected leader, sending a full tablet report...
I20260928 09:04:44.507818 5484 consensus_queue.cc:260] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.508068 5484 ts_tablet_manager.cc:1428] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:04:44.508121 5483 tablet_bootstrap.cc:492] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: No bootstrap required, opened a new log
I20260928 09:04:44.508122 5337 heartbeater.cc:503] Master 127.0.98.62:43879 was elected leader, sending a full tablet report...
I20260928 09:04:44.508178 5483 ts_tablet_manager.cc:1397] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 09:04:44.510232 5483 raft_consensus.cc:348] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.510352 5483 raft_consensus.cc:374] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:44.510373 5483 raft_consensus.cc:729] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88494efbeba4cea82cd074290ef23ff, State: Initialized, Role: FOLLOWER
I20260928 09:04:44.510481 5483 consensus_queue.cc:260] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.510718 5483 ts_tablet_manager.cc:1428] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 09:04:44.510737 5206 heartbeater.cc:503] Master 127.0.98.62:43879 was elected leader, sending a full tablet report...
W20260928 09:04:44.616319 5338 tablet.cc:2369] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:44.721364 5490 raft_consensus.cc:482] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:44.721529 5490 raft_consensus.cc:504] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.721932 5490 leader_election.cc:288] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ed25957c9554e2a8b270503793f88af (127.0.98.2:42493), 7a5fb7b9855b4dd4b39617bbb7261137 (127.0.98.3:41623)
I20260928 09:04:44.723363 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.722135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42493 (local address 127.0.98.1:60587)
0928 09:04:44.722209 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.722224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.722286 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.722456 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.722458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.722460 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.722959 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.722959 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.722968 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.722981 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.723196 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.723197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.723197 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.723276 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.723292 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 09:04:44.723364 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.722240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60587 (local address 127.0.98.2:42493)
0928 09:04:44.722283 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.722285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.722296 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:44.722365 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.722367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.722372 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.722452 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.723076 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.723077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.723077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.723081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.723119 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.723124 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.723190 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.723282 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.723299 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:44.723470 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.722135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.98.1:33189)
0928 09:04:44.722351 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.722360 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:44.722445 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:44.722592 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:44.722593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:44.722596 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:44.723164 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:04:44.723164 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:44.723171 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:04:44.723184 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:44.723331 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:44.723332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:44.723332 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 09:04:44.723376 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 09:04:44.723390 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":54,"thread_start_us":92,"threads_started":1}
I20260928 09:04:44.723498 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.722261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33189 (local address 127.0.98.3:41623)
0928 09:04:44.722304 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.722306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.722440 (+ 134us) server_negotiation.cc:407] Connection header received
0928 09:04:44.722504 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.722505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.722512 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.722562 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.723243 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.723245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.723245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.723250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.723265 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.723269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.723320 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.723386 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.723399 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:44.723765 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd99e096f9e4f59b493a298ab4f2746" candidate_uuid: "c88494efbeba4cea82cd074290ef23ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed25957c9554e2a8b270503793f88af" is_pre_election: true
I20260928 09:04:44.723922 5291 raft_consensus.cc:2454] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88494efbeba4cea82cd074290ef23ff in term 0.
I20260928 09:04:44.723852 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd99e096f9e4f59b493a298ab4f2746" candidate_uuid: "c88494efbeba4cea82cd074290ef23ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" is_pre_election: true
I20260928 09:04:44.724027 5422 raft_consensus.cc:2454] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88494efbeba4cea82cd074290ef23ff in term 0.
I20260928 09:04:44.724156 5096 leader_election.cc:302] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed25957c9554e2a8b270503793f88af, c88494efbeba4cea82cd074290ef23ff; no voters:
I20260928 09:04:44.724316 5490 raft_consensus.cc:2781] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:44.724388 5490 raft_consensus.cc:482] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:44.724423 5490 raft_consensus.cc:3037] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:44.725132 5490 raft_consensus.cc:504] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.725291 5490 leader_election.cc:288] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [CANDIDATE]: Term 1 election: Requested vote from peers 1ed25957c9554e2a8b270503793f88af (127.0.98.2:42493), 7a5fb7b9855b4dd4b39617bbb7261137 (127.0.98.3:41623)
I20260928 09:04:44.725411 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd99e096f9e4f59b493a298ab4f2746" candidate_uuid: "c88494efbeba4cea82cd074290ef23ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed25957c9554e2a8b270503793f88af"
I20260928 09:04:44.725445 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd99e096f9e4f59b493a298ab4f2746" candidate_uuid: "c88494efbeba4cea82cd074290ef23ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5fb7b9855b4dd4b39617bbb7261137"
I20260928 09:04:44.725505 5291 raft_consensus.cc:3037] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:44.725529 5422 raft_consensus.cc:3037] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:44.726392 5291 raft_consensus.cc:2454] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88494efbeba4cea82cd074290ef23ff in term 1.
I20260928 09:04:44.726392 5422 raft_consensus.cc:2454] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88494efbeba4cea82cd074290ef23ff in term 1.
I20260928 09:04:44.726581 5095 leader_election.cc:302] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a5fb7b9855b4dd4b39617bbb7261137, c88494efbeba4cea82cd074290ef23ff; no voters:
I20260928 09:04:44.726677 5490 raft_consensus.cc:2781] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:44.726837 5490 raft_consensus.cc:686] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [term 1 LEADER]: Becoming Leader. State: Replica: c88494efbeba4cea82cd074290ef23ff, State: Running, Role: LEADER
I20260928 09:04:44.726966 5490 consensus_queue.cc:237] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260928 09:04:44.727846 5018 catalog_manager.cc:5495] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff reported cstate change: term changed from 0 to 1, leader changed from <none> to c88494efbeba4cea82cd074290ef23ff (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c88494efbeba4cea82cd074290ef23ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c88494efbeba4cea82cd074290ef23ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } health_report { overall_health: UNKNOWN } } }
W20260928 09:04:44.731702 5469 tablet.cc:2369] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:04:44.751067 5207 tablet.cc:2369] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:44.785060 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.783836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.1:37973)
0928 09:04:44.783903 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.783905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.784009 (+ 104us) server_negotiation.cc:407] Connection header received
0928 09:04:44.784066 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.784067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.784081 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.784124 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.784840 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.784842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.784842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.784846 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.784861 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.784865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.784916 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.784990 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.785002 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:04:44.786988 5291 raft_consensus.cc:1261] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 1 FOLLOWER]: Refusing update from remote peer c88494efbeba4cea82cd074290ef23ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:44.787165 5422 raft_consensus.cc:1261] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 1 FOLLOWER]: Refusing update from remote peer c88494efbeba4cea82cd074290ef23ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:44.787339 5490 consensus_queue.cc:1035] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed25957c9554e2a8b270503793f88af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:44.787479 5492 consensus_queue.cc:1035] T 5dd99e096f9e4f59b493a298ab4f2746 P c88494efbeba4cea82cd074290ef23ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5fb7b9855b4dd4b39617bbb7261137" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:44.790340 5496 mvcc.cc:205] Tried to move back new op lower bound from 7334241422484860928 to 7334241422242340864. Current Snapshot: MvccSnapshot[applied={T|T < 7334241422484860928}]
I20260928 09:04:44.792567 5018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:44.792671 5018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:44.793375 5018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6983868c5e946a7aa1227c0fefb1b41: Sending DeleteTablet for 3 replicas of tablet 5dd99e096f9e4f59b493a298ab4f2746
I20260928 09:04:44.793771 5402 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd99e096f9e4f59b493a298ab4f2746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:44 UTC) from {username='slave'} at 127.0.0.1:58382
I20260928 09:04:44.793701 5271 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd99e096f9e4f59b493a298ab4f2746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:44 UTC) from {username='slave'} at 127.0.0.1:53524
I20260928 09:04:44.794018 5506 tablet_replica.cc:330] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: stopping tablet replica
I20260928 09:04:44.794050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5078
I20260928 09:04:44.794116 5506 raft_consensus.cc:2229] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:44.794211 5506 raft_consensus.cc:2258] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:44.795188 5506 ts_tablet_manager.cc:1905] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:44.796018 5505 tablet_replica.cc:330] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: stopping tablet replica
I20260928 09:04:44.796151 5505 raft_consensus.cc:2229] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:44.796289 5505 raft_consensus.cc:2258] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:44.797187 5506 ts_tablet_manager.cc:1918] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:44.797277 5506 log.cc:1196] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/5dd99e096f9e4f59b493a298ab4f2746
I20260928 09:04:44.797600 5506 ts_tablet_manager.cc:1939] T 5dd99e096f9e4f59b493a298ab4f2746 P 1ed25957c9554e2a8b270503793f88af: Deleting consensus metadata
I20260928 09:04:44.798008 5005 catalog_manager.cc:4841] TS 1ed25957c9554e2a8b270503793f88af (127.0.98.2:42493): tablet 5dd99e096f9e4f59b493a298ab4f2746 (table test-table [id=1e24906f1f904d59b2e9f7aeebafb376]) successfully deleted
W20260928 09:04:44.799592 5003 connection.cc:535] client connection to 127.0.98.1:37973 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 09:04:44.799702 5003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:04:44.800076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5209
I20260928 09:04:44.800096 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.799965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:51910)
0928 09:04:44.800022 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.800062 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 09:04:44.800609 5003 catalog_manager.cc:4584] TS c88494efbeba4cea82cd074290ef23ff (127.0.98.1:37973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dd99e096f9e4f59b493a298ab4f2746: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
I20260928 09:04:44.805155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5340
I20260928 09:04:44.805182 5505 ts_tablet_manager.cc:1905] T 5dd99e096f9e4f59b493a298ab4f2746 P 7a5fb7b9855b4dd4b39617bbb7261137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:44.809229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 4987
I20260928 09:04:44.809273 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.809142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.0.1:58396)
0928 09:04:44.809201 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:44.809234 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
2026-09-28T09:04:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (865 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 09:04:44.829166 392 mini_oidc.cc:152] Starting JWKS server
I20260928 09:04:44.830385 392 webserver.cc:460] Webserver started at https://127.0.0.1:42615/ using document root <none> and password file <none>
I20260928 09:04:44.830510 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 09:04:44.830695 392 webserver.cc:460] Webserver started at http://127.0.0.1:44777/ using document root <none> and password file <none>
2026-09-28T09:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:44Z Disabled control of system clock
I20260928 09:04:44.838424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33059
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33059
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42615/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/testchainca.cert with env {}
W20260928 09:04:44.920499 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:44.920713 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:44.920737 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:44.921880 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:44.921919 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:44.921932 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:44.921944 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:44.921957 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 09:04:44.921976 5513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 09:04:44.921988 5513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42615/jwks.json
I20260928 09:04:44.923570 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42615/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:44.923772 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:44.923930 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:44.928966 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.929091 5513 server_base.cc:1034] running on GCE node
W20260928 09:04:44.928962 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:44.928962 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:44.929605 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:44.930030 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:44.931247 5513 hybrid_clock.cc:648] HybridClock initialized: now 1790586284931216 us; error 47 us; skew 500 ppm
I20260928 09:04:44.938210 5513 webserver.cc:460] Webserver started at http://127.0.98.62:38253/ using document root <none> and password file <none>
I20260928 09:04:44.938483 5513 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:44.938560 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:44.938715 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:44.939843 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "2248dfde91624357a806520b1378d001"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "ece5f0ae9c3afaeb5574479a86580b88"
server_key_iv: "3cba4275dc38768d77a9ea516e526638"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.940233 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "2248dfde91624357a806520b1378d001"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "ece5f0ae9c3afaeb5574479a86580b88"
server_key_iv: "3cba4275dc38768d77a9ea516e526638"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.941531 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:44.942390 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:44.942579 5513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:44.942684 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "2248dfde91624357a806520b1378d001"
format_stamp: "Formatted at 2026-09-28 09:04:44 on dist-test-slave-smns"
server_key: "ece5f0ae9c3afaeb5574479a86580b88"
server_key_iv: "3cba4275dc38768d77a9ea516e526638"
server_key_version: "encryptionkey@0"
I20260928 09:04:44.942768 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:44.961658 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:44.961989 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:44.962143 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:44.966217 5513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33059
I20260928 09:04:44.966287 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33059 every 8 connection(s)
I20260928 09:04:44.966645 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:44.967648 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:44.969964 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001: Bootstrap starting.
I20260928 09:04:44.970594 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:44.970904 5580 log.cc:824] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:44.971652 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001: No bootstrap required, opened a new log
I20260928 09:04:44.973132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5513
I20260928 09:04:44.973242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:44.973443 392 external_mini_cluster.cc:1428] Setting key c6cfda84b610d0c17f5e6db0ac7221a2
I20260928 09:04:44.973608 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2248dfde91624357a806520b1378d001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33059 } }
I20260928 09:04:44.973790 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:44.973834 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2248dfde91624357a806520b1378d001, State: Initialized, Role: FOLLOWER
I20260928 09:04:44.973966 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2248dfde91624357a806520b1378d001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33059 } }
I20260928 09:04:44.974052 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:44.974085 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:44.974141 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:44.974673 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2248dfde91624357a806520b1378d001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33059 } }
I20260928 09:04:44.974802 5580 leader_election.cc:302] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2248dfde91624357a806520b1378d001; no voters:
I20260928 09:04:44.974985 5580 leader_election.cc:288] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:44.975163 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:44.975234 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:44.975291 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [term 1 LEADER]: Becoming Leader. State: Replica: 2248dfde91624357a806520b1378d001, State: Running, Role: LEADER
I20260928 09:04:44.975389 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2248dfde91624357a806520b1378d001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33059 } }
I20260928 09:04:44.975690 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2248dfde91624357a806520b1378d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2248dfde91624357a806520b1378d001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33059 } } }
I20260928 09:04:44.975791 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:44.975688 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2248dfde91624357a806520b1378d001. Latest consensus state: current_term: 1 leader_uuid: "2248dfde91624357a806520b1378d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2248dfde91624357a806520b1378d001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33059 } } }
I20260928 09:04:44.976047 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:44.976158 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:44.976689 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:44.978561 5594 catalog_manager.cc:1347] Generated new cluster ID: 3d8b2e4f8fa64353bfea63b7ce0862d5
I20260928 09:04:44.978624 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:44.979641 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:44.974053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.98.62:33059)
0928 09:04:44.974294 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:44.974302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:44.974326 (+ 24us) server_negotiation.cc:407] Connection header received
0928 09:04:44.974429 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:44.974436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:44.974495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:44.974644 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:44.974887 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:44.976404 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:44.977578 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:44.978175 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:44.978302 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:44.979011 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:44.979014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:44.979015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:44.979191 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:44.979221 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:44.979232 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:44.979318 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:44.979390 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 09:04:44.979470 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260928 09:04:44.982497 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:04:44.995373 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:44.996089 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:44.999025 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2248dfde91624357a806520b1378d001: Generated new TSK 0
I20260928 09:04:44.999191 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 09:04:45.030282 5610 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 09:04:45.030393 5610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 09:04:45.030656 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.007603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.0.98.62:33059)
0928 09:04:45.007653 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.007656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.007783 (+ 127us) server_negotiation.cc:407] Connection header received
0928 09:04:45.007923 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.007924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.007947 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.008044 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0928 09:04:45.008203 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.008737 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.009575 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.009784 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.009877 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.010097 (+ 220us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 09:04:45.030283 (+ 20186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 09:04:45.030585 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 09:04:45.030722 5584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 09:04:45.030973 5609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:45.007524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33059 (local address 127.0.0.1:57572)
0928 09:04:45.007707 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:45.007718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:45.007805 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:45.008055 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:45.008056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:45.008065 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 09:04:45.008120 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.008122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.008837 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.008839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.009460 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.009462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.009877 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.009880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.009937 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.009966 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 09:04:45.030494 (+ 20528us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 09:04:45.030612 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 09:04:45.031379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5513
2026-09-28T09:04:45Z chronyd exiting
I20260928 09:04:45.066634 392 test_util.cc:175] -----------------------------------------------
I20260928 09:04:45.066720 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790586261566264-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 09:04:45.067622 392 mini_oidc.cc:152] Starting JWKS server
I20260928 09:04:45.068303 392 webserver.cc:460] Webserver started at https://127.0.0.1:46295/ using document root <none> and password file <none>
I20260928 09:04:45.068347 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 09:04:45.068498 392 webserver.cc:460] Webserver started at http://127.0.0.1:45427/ using document root <none> and password file <none>
2026-09-28T09:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:45Z Disabled control of system clock
I20260928 09:04:45.075938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46217
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46217
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46295/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/testchainca.cert with env {}
W20260928 09:04:45.156355 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:45.156589 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:45.156616 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:45.157835 5616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:45.157874 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:45.157891 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:45.157903 5616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:45.157922 5616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 09:04:45.157935 5616 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 09:04:45.157948 5616 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46295/jwks.json
I20260928 09:04:45.159451 5616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46295/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:45.159655 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:45.159822 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:45.164945 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:45.164925 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:45.164916 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:45.165323 5616 server_base.cc:1034] running on GCE node
I20260928 09:04:45.165635 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:45.165982 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:45.167181 5616 hybrid_clock.cc:648] HybridClock initialized: now 1790586285167167 us; error 27 us; skew 500 ppm
I20260928 09:04:45.168615 5616 webserver.cc:460] Webserver started at http://127.0.98.62:35967/ using document root <none> and password file <none>
I20260928 09:04:45.168906 5616 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:45.168963 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:45.169127 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:45.170336 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "472c9daf69c84042a403fd8f4e6f76ec"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "d7c3e1ef7ba299201486a8e22b31f778"
server_key_iv: "31ef869d1d31adf33a82135237608686"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.170655 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "472c9daf69c84042a403fd8f4e6f76ec"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "d7c3e1ef7ba299201486a8e22b31f778"
server_key_iv: "31ef869d1d31adf33a82135237608686"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.171947 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:04:45.172658 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:45.172910 5616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:45.172982 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "472c9daf69c84042a403fd8f4e6f76ec"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "d7c3e1ef7ba299201486a8e22b31f778"
server_key_iv: "31ef869d1d31adf33a82135237608686"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.173051 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:45.215502 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:45.215891 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:45.216012 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:45.220544 5616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46217
I20260928 09:04:45.220588 5682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46217 every 8 connection(s)
I20260928 09:04:45.220949 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:45.221936 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:45.224228 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec: Bootstrap starting.
I20260928 09:04:45.224798 5683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:45.225093 5683 log.cc:824] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:45.225977 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec: No bootstrap required, opened a new log
I20260928 09:04:45.227605 5683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c9daf69c84042a403fd8f4e6f76ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46217 } }
I20260928 09:04:45.227741 5683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:45.227772 5683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472c9daf69c84042a403fd8f4e6f76ec, State: Initialized, Role: FOLLOWER
I20260928 09:04:45.227984 5683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c9daf69c84042a403fd8f4e6f76ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46217 } }
I20260928 09:04:45.228068 5683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:45.228092 5683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:45.228121 5683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:45.228677 5683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c9daf69c84042a403fd8f4e6f76ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46217 } }
I20260928 09:04:45.228814 5683 leader_election.cc:302] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472c9daf69c84042a403fd8f4e6f76ec; no voters:
I20260928 09:04:45.228998 5683 leader_election.cc:288] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:45.229104 5686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:45.229280 5683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:45.229283 5686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [term 1 LEADER]: Becoming Leader. State: Replica: 472c9daf69c84042a403fd8f4e6f76ec, State: Running, Role: LEADER
I20260928 09:04:45.229431 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c9daf69c84042a403fd8f4e6f76ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46217 } }
I20260928 09:04:45.229847 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472c9daf69c84042a403fd8f4e6f76ec. Latest consensus state: current_term: 1 leader_uuid: "472c9daf69c84042a403fd8f4e6f76ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c9daf69c84042a403fd8f4e6f76ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46217 } } }
I20260928 09:04:45.229933 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:45.229964 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472c9daf69c84042a403fd8f4e6f76ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472c9daf69c84042a403fd8f4e6f76ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46217 } } }
I20260928 09:04:45.230062 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:45.230314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5616
I20260928 09:04:45.230425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:45.230634 392 external_mini_cluster.cc:1428] Setting key fde9cbc55188b30a3eac82c8011bdd52
I20260928 09:04:45.230746 5695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:45.231529 5695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:45.233162 5695 catalog_manager.cc:1347] Generated new cluster ID: 9ab7e66887524d82b7e62bae9b86d91a
I20260928 09:04:45.233218 5695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:45.237094 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.231164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.0.98.62:46217)
0928 09:04:45.231357 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.231367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.231550 (+ 183us) server_negotiation.cc:407] Connection header received
0928 09:04:45.231720 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.231727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.231799 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.231957 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:45.232515 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.233868 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.234957 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.235584 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.235710 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.236391 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:45.236395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:45.236395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:45.236604 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:45.236644 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:45.236658 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:45.236726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:45.236791 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:04:45.236867 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260928 09:04:45.238027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:04:45.252560 5695 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:45.253288 5695 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:45.259181 5695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472c9daf69c84042a403fd8f4e6f76ec: Generated new TSK 0
I20260928 09:04:45.259371 5695 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 09:04:45.282542 5713 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 09:04:45.282642 5713 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 09:04:45.282984 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.268237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.0.98.62:46217)
0928 09:04:45.268279 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.268281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.268380 (+ 99us) server_negotiation.cc:407] Connection header received
0928 09:04:45.268421 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.268422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.268434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.268516 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0928 09:04:45.268684 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.269485 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.270298 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.270623 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.270694 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.270863 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 09:04:45.282740 (+ 11877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 09:04:45.282897 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 09:04:45.283037 5705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 09:04:45.283077 5712 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:45.268114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46217 (local address 127.0.0.1:54062)
0928 09:04:45.268314 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:45.268323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:45.268372 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:45.268550 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:45.268552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:45.268554 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 09:04:45.268614 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.268616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.269598 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.269599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.270211 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.270213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.270704 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.270707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.270747 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.270751 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 09:04:45.282863 (+ 12112us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 09:04:45.282953 (+ 90us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46217: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260928 09:04:45.283514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5616
2026-09-28T09:04:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (253 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 09:04:45.320962 392 mini_oidc.cc:152] Starting JWKS server
I20260928 09:04:45.321733 392 webserver.cc:460] Webserver started at https://127.0.0.1:34981/ using document root <none> and password file <none>
I20260928 09:04:45.321777 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 09:04:45.321959 392 webserver.cc:460] Webserver started at http://127.0.0.1:40363/ using document root <none> and password file <none>
2026-09-28T09:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:45Z Disabled control of system clock
I20260928 09:04:45.329790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 09:04:45.410028 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:45.410252 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:45.410274 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:45.411475 5719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:45.411520 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:45.411533 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:45.411546 5719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:45.411559 5719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 09:04:45.411571 5719 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 09:04:45.411583 5719 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34981/jwks.json
I20260928 09:04:45.412931 5719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:45.413142 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:45.413439 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:45.418743 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:45.418735 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:45.418776 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:45.418952 5719 server_base.cc:1034] running on GCE node
I20260928 09:04:45.419373 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:45.419682 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:45.420886 5719 hybrid_clock.cc:648] HybridClock initialized: now 1790586285420882 us; error 40 us; skew 500 ppm
I20260928 09:04:45.422289 5719 webserver.cc:460] Webserver started at http://127.0.98.62:41351/ using document root <none> and password file <none>
I20260928 09:04:45.422536 5719 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:45.422611 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:45.422770 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:45.423925 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "897acc83756b4bd7b926223b60f54005"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "0cf7a7a60092b9c9afb8001319d8aee2"
server_key_iv: "271e0004664184a5c3db016e606b04bc"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.424306 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "897acc83756b4bd7b926223b60f54005"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "0cf7a7a60092b9c9afb8001319d8aee2"
server_key_iv: "271e0004664184a5c3db016e606b04bc"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.425673 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:04:45.426469 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:45.426649 5719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:45.426772 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "897acc83756b4bd7b926223b60f54005"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "0cf7a7a60092b9c9afb8001319d8aee2"
server_key_iv: "271e0004664184a5c3db016e606b04bc"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.426908 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:45.442929 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:45.443259 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:45.443419 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:45.447531 5719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41303
I20260928 09:04:45.447566 5785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41303 every 8 connection(s)
I20260928 09:04:45.447913 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:45.448588 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:45.451323 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005: Bootstrap starting.
I20260928 09:04:45.451865 5786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:45.452105 5786 log.cc:824] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:45.452805 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005: No bootstrap required, opened a new log
I20260928 09:04:45.454171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5719
I20260928 09:04:45.454264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:45.454430 392 external_mini_cluster.cc:1428] Setting key 26dd8d8c2ab893e385922a3933f284c8
I20260928 09:04:45.454788 5786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897acc83756b4bd7b926223b60f54005" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260928 09:04:45.454969 5786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:45.455019 5786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897acc83756b4bd7b926223b60f54005, State: Initialized, Role: FOLLOWER
I20260928 09:04:45.455155 5786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897acc83756b4bd7b926223b60f54005" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260928 09:04:45.455235 5786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:45.455256 5786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:45.455318 5786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:45.455997 5786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897acc83756b4bd7b926223b60f54005" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260928 09:04:45.456084 5786 leader_election.cc:302] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897acc83756b4bd7b926223b60f54005; no voters:
I20260928 09:04:45.456260 5786 leader_election.cc:288] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:45.456287 5791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:45.456393 5791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [term 1 LEADER]: Becoming Leader. State: Replica: 897acc83756b4bd7b926223b60f54005, State: Running, Role: LEADER
I20260928 09:04:45.456473 5791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897acc83756b4bd7b926223b60f54005" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260928 09:04:45.456530 5786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:45.456770 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897acc83756b4bd7b926223b60f54005. Latest consensus state: current_term: 1 leader_uuid: "897acc83756b4bd7b926223b60f54005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897acc83756b4bd7b926223b60f54005" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } } }
I20260928 09:04:45.456851 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:45.456784 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897acc83756b4bd7b926223b60f54005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897acc83756b4bd7b926223b60f54005" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } } }
I20260928 09:04:45.456993 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:45.457065 5795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:45.457638 5795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:45.459322 5795 catalog_manager.cc:1347] Generated new cluster ID: 497b70dcc51e47c690d982632d03a805
I20260928 09:04:45.459378 5795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:45.460795 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.454980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.0.98.62:41303)
0928 09:04:45.455194 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.455201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.455251 (+ 50us) server_negotiation.cc:407] Connection header received
0928 09:04:45.455393 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.455400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.455468 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.455576 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:45.455820 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.457583 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.458733 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.459292 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.459412 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.460116 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:45.460120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:45.460120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:45.460295 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:45.460328 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:45.460338 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:45.460418 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:45.460510 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:04:45.460593 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260928 09:04:45.461699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:04:45.499816 5795 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:45.500386 5795 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:45.509425 5795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897acc83756b4bd7b926223b60f54005: Generated new TSK 0
I20260928 09:04:45.509658 5795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:04:45.523351 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.519601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.0.98.62:41303)
0928 09:04:45.519672 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.519674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.519797 (+ 123us) server_negotiation.cc:407] Connection header received
0928 09:04:45.519879 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.519880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.519893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.519966 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 09:04:45.520154 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.520649 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.521491 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.521766 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.521850 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.522106 (+ 256us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 09:04:45.523136 (+ 1030us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 09:04:45.523273 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 09:04:45.523416 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 09:04:45.523515 5815 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:45.519524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41303 (local address 127.0.0.1:45880)
0928 09:04:45.519719 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:45.519728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:45.519815 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:45.520004 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:45.520005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:45.520007 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 09:04:45.520078 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.520080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.520750 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.520751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.521369 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.521371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.521866 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.521874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.521938 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.521948 (+ 10us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 09:04:45.523293 (+ 1345us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 09:04:45.523395 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":119,"threads_started":1}
I20260928 09:04:45.524288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5719
E20260928 09:04:45.535979 5816 webserver.cc:533] Webserver: sslize failed:
E20260928 09:04:45.536052 5816 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T09:04:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T09:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:45Z Disabled control of system clock
I20260928 09:04:45.569490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43407
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43407
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/testchainca.cert with env {}
W20260928 09:04:45.649149 5820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:45.649348 5820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:45.649370 5820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:45.650501 5820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:45.650538 5820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:45.650552 5820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:45.650569 5820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:45.650583 5820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 09:04:45.650594 5820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 09:04:45.650606 5820 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 09:04:45.652024 5820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:45.652235 5820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:45.652395 5820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:45.657301 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:45.657297 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:45.657413 5820 server_base.cc:1034] running on GCE node
W20260928 09:04:45.657348 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:45.657799 5820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:45.658133 5820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:45.659274 5820 hybrid_clock.cc:648] HybridClock initialized: now 1790586285659254 us; error 37 us; skew 500 ppm
I20260928 09:04:45.660431 5820 webserver.cc:460] Webserver started at http://127.0.98.62:44603/ using document root <none> and password file <none>
I20260928 09:04:45.660679 5820 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:45.660725 5820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:45.660898 5820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:45.662007 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "5923a5b61be64641887c35168b085559"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "17371e5a9af35ef535d34030fbd88364"
server_key_iv: "68a4c94eb30e78c92641d851c187fad1"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.662364 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "5923a5b61be64641887c35168b085559"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "17371e5a9af35ef535d34030fbd88364"
server_key_iv: "68a4c94eb30e78c92641d851c187fad1"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.663717 5820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:45.664506 5834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:45.664726 5820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:45.664796 5820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "5923a5b61be64641887c35168b085559"
format_stamp: "Formatted at 2026-09-28 09:04:45 on dist-test-slave-smns"
server_key: "17371e5a9af35ef535d34030fbd88364"
server_key_iv: "68a4c94eb30e78c92641d851c187fad1"
server_key_version: "encryptionkey@0"
I20260928 09:04:45.664889 5820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:45.683087 5820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:45.683430 5820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:45.683593 5820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:45.687944 5820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43407
I20260928 09:04:45.688021 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43407 every 8 connection(s)
I20260928 09:04:45.688365 5820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:45.689361 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:45.691731 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559: Bootstrap starting.
I20260928 09:04:45.692323 5887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:45.692627 5887 log.cc:824] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:45.693346 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559: No bootstrap required, opened a new log
I20260928 09:04:45.693727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5820
I20260928 09:04:45.693825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:45.694021 392 external_mini_cluster.cc:1428] Setting key 3d1d3470b0d974df1ff96a1ad1f2a94e
I20260928 09:04:45.695688 5887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5923a5b61be64641887c35168b085559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43407 } }
I20260928 09:04:45.695858 5887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:45.695895 5887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5923a5b61be64641887c35168b085559, State: Initialized, Role: FOLLOWER
I20260928 09:04:45.696050 5887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5923a5b61be64641887c35168b085559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43407 } }
I20260928 09:04:45.696147 5887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:45.696195 5887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:45.696235 5887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:45.696971 5887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5923a5b61be64641887c35168b085559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43407 } }
I20260928 09:04:45.697082 5887 leader_election.cc:302] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5923a5b61be64641887c35168b085559; no voters:
I20260928 09:04:45.697227 5887 leader_election.cc:288] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:45.697260 5892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:45.697412 5892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [term 1 LEADER]: Becoming Leader. State: Replica: 5923a5b61be64641887c35168b085559, State: Running, Role: LEADER
I20260928 09:04:45.697469 5887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:45.697497 5892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5923a5b61be64641887c35168b085559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43407 } }
I20260928 09:04:45.697824 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5923a5b61be64641887c35168b085559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5923a5b61be64641887c35168b085559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43407 } } }
I20260928 09:04:45.697924 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:45.697824 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5923a5b61be64641887c35168b085559. Latest consensus state: current_term: 1 leader_uuid: "5923a5b61be64641887c35168b085559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5923a5b61be64641887c35168b085559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43407 } } }
I20260928 09:04:45.698047 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:45.699003 5899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:45.699532 5899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:45.701334 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.694574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.0.98.62:43407)
0928 09:04:45.694802 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.694812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.694847 (+ 35us) server_negotiation.cc:407] Connection header received
0928 09:04:45.695647 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.695656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.695738 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.695894 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:45.696204 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.698051 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.698887 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.699629 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.699762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.700615 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:45.700619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:45.700619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:04:45.700805 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:45.700837 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:04:45.700847 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:45.700949 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:45.701055 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 09:04:45.701135 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260928 09:04:45.701648 5899 catalog_manager.cc:1347] Generated new cluster ID: f7bbab5af106448596d94ab83ea0c65e
I20260928 09:04:45.701709 5899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:45.702468 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:04:45.709909 5899 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:45.710652 5899 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:45.718310 5899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5923a5b61be64641887c35168b085559: Generated new TSK 0
I20260928 09:04:45.718510 5899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:04:45.864728 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:45.724619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.0.98.62:43407)
0928 09:04:45.724665 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:45.724667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:45.724749 (+ 82us) server_negotiation.cc:407] Connection header received
0928 09:04:45.724812 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:45.724812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:45.724833 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:45.724893 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 09:04:45.725045 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.725506 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.726956 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.727134 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.727185 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.727386 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 09:04:45.864507 (+137121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 09:04:45.864667 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45628: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 09:04:45.864818 5891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45628: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 09:04:45.864913 5916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:45.724524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43407 (local address 127.0.0.1:45628)
0928 09:04:45.724669 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:45.724680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:45.724746 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:45.724919 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:45.724919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:45.724921 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 09:04:45.724980 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.724982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.725859 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.725860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.726856 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:45.726859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:45.727215 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:45.727220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:45.727280 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:45.727287 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 09:04:45.864655 (+137368us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 09:04:45.864785 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43407: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":100,"threads_started":1}
I20260928 09:04:45.865250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5820
2026-09-28T09:04:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (320 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:46 dist-test-slave-smns krb5kdc[5928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:46 dist-test-slave-smns krb5kdc[5928](info): set up 2 sockets
Sep 28 09:04:46 dist-test-slave-smns krb5kdc[5928](info): commencing operation
krb5kdc: starting...
W20260928 09:04:47.093926 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:47.221251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:47.301185 5941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:47.301448 5941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:47.301488 5941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:47.302677 5941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:47.302760 5941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:47.302785 5941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:47.302814 5941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:47.302842 5941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:47.304360 5941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:47.304632 5941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:47.304857 5941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:47.309747 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:47.309864 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:47.309777 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:47.309880 5941 server_base.cc:1034] running on GCE node
I20260928 09:04:47.310233 5941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:47.310530 5941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:47.311703 5941 hybrid_clock.cc:648] HybridClock initialized: now 1790586287311681 us; error 39 us; skew 500 ppm
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:47.314251 5941 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:47.314719 5941 webserver.cc:460] Webserver started at http://127.0.98.62:34685/ using document root <none> and password file <none>
I20260928 09:04:47.314950 5941 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:47.315021 5941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:47.315188 5941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:47.316318 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "7f9d63c60884c341f0c800723d17f69a"
server_key_iv: "fa0bb374700efb578484ef2ceafc2295"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.316672 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "7f9d63c60884c341f0c800723d17f69a"
server_key_iv: "fa0bb374700efb578484ef2ceafc2295"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.318068 5941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:04:47.318902 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.319077 5941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:47.319180 5941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "7f9d63c60884c341f0c800723d17f69a"
server_key_iv: "fa0bb374700efb578484ef2ceafc2295"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.319262 5941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:47.337273 5941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:47.339764 5941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:47.339944 5941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:47.344074 5941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33159
I20260928 09:04:47.344112 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33159 every 8 connection(s)
I20260928 09:04:47.344609 5941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:47.345270 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:47.346062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 5941
I20260928 09:04:47.346179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:47.346376 392 external_mini_cluster.cc:1428] Setting key 55b749ec22aee96bdae22a58173ddcb0
I20260928 09:04:47.348558 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c: Bootstrap starting.
I20260928 09:04:47.349169 6009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:47.349459 6009 log.cc:824] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:47.350229 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c: No bootstrap required, opened a new log
I20260928 09:04:47.352792 6009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33159 } }
I20260928 09:04:47.352928 6009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:47.352949 6009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f73b4ef4e4d4403bfbc8ec9cb0ff50c, State: Initialized, Role: FOLLOWER
I20260928 09:04:47.353085 6009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33159 } }
I20260928 09:04:47.353161 6009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:47.353184 6009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:47.353257 6009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 0 FOLLOWER]: Advancing to term 1
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:47.353891 6009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33159 } }
I20260928 09:04:47.354015 6009 leader_election.cc:302] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f73b4ef4e4d4403bfbc8ec9cb0ff50c; no voters:
I20260928 09:04:47.354195 6009 leader_election.cc:288] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:47.354223 6014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:47.354388 6014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [term 1 LEADER]: Becoming Leader. State: Replica: 0f73b4ef4e4d4403bfbc8ec9cb0ff50c, State: Running, Role: LEADER
I20260928 09:04:47.354540 6009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:47.354529 6014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33159 } }
I20260928 09:04:47.354866 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f73b4ef4e4d4403bfbc8ec9cb0ff50c. Latest consensus state: current_term: 1 leader_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33159 } } }
I20260928 09:04:47.354943 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f73b4ef4e4d4403bfbc8ec9cb0ff50c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33159 } } }
I20260928 09:04:47.355006 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:47.356199 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c [sys.catalog]: This master's current role is: LEADER
W20260928 09:04:47.356375 6028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:04:47.356428 6028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:04:47.356614 6029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:47.356825 6029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:47.357721 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:47.346943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.0.98.62:33159)
0928 09:04:47.347122 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:47.347134 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:47.347160 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:04:47.347577 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:47.347584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:47.347628 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:47.347737 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:47.348521 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.349557 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.350567 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.350957 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.351039 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:47.353927 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:47.353932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:47.353932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:47.353945 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:47.356022 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:47.356425 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:47.356428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:47.356429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:47.356540 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:47.356775 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:47.356776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:47.356776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:47.357073 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:47.357213 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:47.357365 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:04:47.357509 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260928 09:04:47.358355 6029 catalog_manager.cc:1347] Generated new cluster ID: aa1b1c91ff6c44a694824e04400d83aa
I20260928 09:04:47.358403 6029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:47.375088 6029 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:47.375633 6029 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:47.388933 6029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f73b4ef4e4d4403bfbc8ec9cb0ff50c: Generated new TSK 0
I20260928 09:04:47.389183 6029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:47.429401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33159
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:47.520213 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:47.520457 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:47.520496 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:47.521740 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:47.521811 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:47.521916 6035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:47.523522 6035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:47.523720 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:47.523897 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:47.529114 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:47.529093 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:47.529220 6035 server_base.cc:1034] running on GCE node
W20260928 09:04:47.529079 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:47.529619 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:47.529843 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:47.530985 6035 hybrid_clock.cc:648] HybridClock initialized: now 1790586287530973 us; error 28 us; skew 500 ppm
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:47.533663 6035 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:47.534096 6035 webserver.cc:460] Webserver started at http://127.0.98.1:43051/ using document root <none> and password file <none>
I20260928 09:04:47.534358 6035 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:47.534433 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:47.534583 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:47.535671 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "b0b7da7aa719443ab05ac1a73f7747ca"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "2b09effe4c9f639f24c9bf8878cf36fa"
server_key_iv: "51cb65ca4d773a5de80f6757ee5fc7cf"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.536057 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b0b7da7aa719443ab05ac1a73f7747ca"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "2b09effe4c9f639f24c9bf8878cf36fa"
server_key_iv: "51cb65ca4d773a5de80f6757ee5fc7cf"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.537473 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:04:47.538380 6050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.538597 6035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:47.538707 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "b0b7da7aa719443ab05ac1a73f7747ca"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "2b09effe4c9f639f24c9bf8878cf36fa"
server_key_iv: "51cb65ca4d773a5de80f6757ee5fc7cf"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.538802 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:47.556969 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:47.558972 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:47.559156 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:47.559419 6035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:47.559738 6035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:47.559801 6035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.559839 6035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:47.559885 6035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.565651 6035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44315
I20260928 09:04:47.565714 6163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44315 every 8 connection(s)
I20260928 09:04:47.566071 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:47.574927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6035
I20260928 09:04:47.575067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:47.575286 392 external_mini_cluster.cc:1428] Setting key 0123c5d466b549b50ee395a252e51cd0
I20260928 09:04:47.575428 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:47.566883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35987 (local address 127.0.98.62:33159)
0928 09:04:47.566989 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:47.566991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:47.567147 (+ 156us) server_negotiation.cc:407] Connection header received
0928 09:04:47.568083 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:47.568084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:47.568094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:47.568149 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:47.568737 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.569246 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.570255 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.570450 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.570505 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:47.572717 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:47.572719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:47.572719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:47.572726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:47.574071 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:47.574468 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:47.574469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:47.574469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:47.574517 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:47.574733 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:47.574734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:47.574734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:47.574835 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:47.574948 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:47.575245 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 09:04:47.575334 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 09:04:47.575675 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:47.566823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33159 (local address 127.0.98.1:35987)
0928 09:04:47.567013 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:47.567050 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:47.567990 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:47.568207 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:47.568210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:47.568465 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:47.568661 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:47.568666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.569344 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.569345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:47.570182 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:47.570185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.570540 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.570541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:47.570639 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:47.571216 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:47.571225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:47.572621 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:47.574172 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:47.574178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:47.574180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:47.574401 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:47.574580 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:47.574581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:47.574581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:47.574677 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:47.574917 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:47.574925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:47.575108 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 09:04:47.575224 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 09:04:47.575382 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260928 09:04:47.576099 6164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33159
I20260928 09:04:47.576218 6164 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:47.576388 6164 heartbeater.cc:511] Master 127.0.98.62:33159 requested a full tablet report, sending...
I20260928 09:04:47.576952 5973 ts_manager.cc:194] Registered new tserver with Master: b0b7da7aa719443ab05ac1a73f7747ca (127.0.98.1:44315)
I20260928 09:04:47.577603 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35987
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:47.600936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33159
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:47.681957 6169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:47.682217 6169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:47.682260 6169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:47.683447 6169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:47.683531 6169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:47.683579 6169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:47.685194 6169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:47.685482 6169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:47.685747 6169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:47.690933 6175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:47.690908 6174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:47.690917 6177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:47.691246 6169 server_base.cc:1034] running on GCE node
I20260928 09:04:47.691645 6169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:47.691920 6169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:47.693125 6169 hybrid_clock.cc:648] HybridClock initialized: now 1790586287693101 us; error 42 us; skew 500 ppm
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:47.696009 6169 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:47.696473 6169 webserver.cc:460] Webserver started at http://127.0.98.2:35435/ using document root <none> and password file <none>
I20260928 09:04:47.696705 6169 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:47.696779 6169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:47.696947 6169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:47.698100 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae511b60519843d1aca321d2096fa8c4"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "34225017a0eddc0ae7c4f92d18234cd7"
server_key_iv: "0a20f9a764993fe7bc4173b734867503"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.698491 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae511b60519843d1aca321d2096fa8c4"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "34225017a0eddc0ae7c4f92d18234cd7"
server_key_iv: "0a20f9a764993fe7bc4173b734867503"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.699911 6169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:47.700671 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.700906 6169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:47.701014 6169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "ae511b60519843d1aca321d2096fa8c4"
format_stamp: "Formatted at 2026-09-28 09:04:47 on dist-test-slave-smns"
server_key: "34225017a0eddc0ae7c4f92d18234cd7"
server_key_iv: "0a20f9a764993fe7bc4173b734867503"
server_key_version: "encryptionkey@0"
I20260928 09:04:47.701102 6169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:47.733316 6169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:47.735996 6169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:47.736167 6169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:47.736421 6169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:47.736768 6169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:47.736829 6169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.736868 6169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:47.736896 6169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:47.743113 6169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32999
I20260928 09:04:47.743158 6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32999 every 8 connection(s)
I20260928 09:04:47.743476 6169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:47.745836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6169
I20260928 09:04:47.745949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:47.746143 392 external_mini_cluster.cc:1428] Setting key 1e087a3d8ac7f620cdeed307320966fd
Sep 28 09:04:47 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:47.754554 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:47.744223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48175 (local address 127.0.98.62:33159)
0928 09:04:47.744286 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:47.744288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:47.744516 (+ 228us) server_negotiation.cc:407] Connection header received
0928 09:04:47.745449 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:47.745450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:47.745459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:47.745521 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:47.746447 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.747063 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.748234 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.748426 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.748484 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:47.751081 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:47.751083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:47.751083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:47.751092 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:47.752845 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:47.753260 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:47.753262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:47.753262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:47.753338 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:47.753612 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:47.753613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:47.753614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:47.753769 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:47.753847 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:47.754243 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 09:04:47.754364 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:04:47.754693 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:47.744213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33159 (local address 127.0.98.2:48175)
0928 09:04:47.744378 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:47.744421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:47.745350 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:47.745660 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:47.745665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:47.746081 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:47.746373 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:47.746379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.747174 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.747176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:47.748140 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:47.748143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:47.748515 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:47.748517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:47.748668 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:47.749274 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:47.749282 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:47.750975 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:47.752950 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:47.752956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:47.752958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:47.753176 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:47.753412 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:47.753413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:47.753413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:47.753508 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:47.753840 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:47.753845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:47.754053 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 09:04:47.754176 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 09:04:47.754361 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:04:47.755261 6298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33159
I20260928 09:04:47.755364 6298 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:47.755576 6298 heartbeater.cc:511] Master 127.0.98.62:33159 requested a full tablet report, sending...
I20260928 09:04:47.756003 5973 ts_manager.cc:194] Registered new tserver with Master: ae511b60519843d1aca321d2096fa8c4 (127.0.98.2:32999)
I20260928 09:04:47.756526 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260928 09:04:47.933761 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.187s user 0.001s sys 0.002s
I20260928 09:04:47.934098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33159
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:48.017686 6303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:48.017954 6303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:48.017997 6303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:48.019230 6303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:48.019310 6303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:48.019359 6303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:48.021041 6303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:48.021335 6303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:48.021687 6303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:48.027004 6311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:48.026969 6308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:48.026969 6309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:48.027185 6303 server_base.cc:1034] running on GCE node
I20260928 09:04:48.027500 6303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:48.027710 6303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:48.028887 6303 hybrid_clock.cc:648] HybridClock initialized: now 1790586288028870 us; error 27 us; skew 500 ppm
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.031469 6303 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:48.031944 6303 webserver.cc:460] Webserver started at http://127.0.98.3:46469/ using document root <none> and password file <none>
I20260928 09:04:48.032128 6303 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:48.032169 6303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:48.032260 6303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:48.033275 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "5d16d8e68b4e41aaa886d0e1c2a7af10"
format_stamp: "Formatted at 2026-09-28 09:04:48 on dist-test-slave-smns"
server_key: "94b8f6c11a04edf6bd8ff8ad521ef074"
server_key_iv: "006fe5663937b9426c96f896646010f5"
server_key_version: "encryptionkey@0"
I20260928 09:04:48.033667 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5d16d8e68b4e41aaa886d0e1c2a7af10"
format_stamp: "Formatted at 2026-09-28 09:04:48 on dist-test-slave-smns"
server_key: "94b8f6c11a04edf6bd8ff8ad521ef074"
server_key_iv: "006fe5663937b9426c96f896646010f5"
server_key_version: "encryptionkey@0"
I20260928 09:04:48.034964 6303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:48.035666 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:48.035868 6303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:48.035940 6303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "5d16d8e68b4e41aaa886d0e1c2a7af10"
format_stamp: "Formatted at 2026-09-28 09:04:48 on dist-test-slave-smns"
server_key: "94b8f6c11a04edf6bd8ff8ad521ef074"
server_key_iv: "006fe5663937b9426c96f896646010f5"
server_key_version: "encryptionkey@0"
I20260928 09:04:48.035996 6303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:48.043893 6303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:48.045693 6303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:48.045806 6303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:48.045992 6303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:48.046241 6303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:48.046270 6303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:48.046293 6303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:48.046305 6303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:48.052271 6303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45471
I20260928 09:04:48.052312 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45471 every 8 connection(s)
I20260928 09:04:48.052642 6303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:48.058662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6303
I20260928 09:04:48.058770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:48.058996 392 external_mini_cluster.cc:1428] Setting key be92dceb302ec7dc97a5d2877834da5e
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:48.062716 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:48.053427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46197 (local address 127.0.98.62:33159)
0928 09:04:48.053503 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.053505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.053812 (+ 307us) server_negotiation.cc:407] Connection header received
0928 09:04:48.054756 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.054828 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.054841 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.054938 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.055591 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.056178 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.057162 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.057355 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.057414 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:48.060027 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:48.060028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:48.060028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:48.060035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:48.061493 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.061930 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.061931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.061932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.061984 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.062176 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.062177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.062177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.062277 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:48.062323 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:48.062592 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 09:04:48.062651 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 09:04:48.062870 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:48.053454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33159 (local address 127.0.98.3:46197)
0928 09:04:48.053665 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:48.053715 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:48.054642 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:48.054991 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:48.054995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:48.055248 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:48.055503 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:48.055508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.056272 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.056273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:48.057075 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:48.057076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.057432 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.057433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:48.057530 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:48.058117 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:48.058126 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:48.059940 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:48.061606 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:48.061611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:48.061613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:48.061848 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:48.062029 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:48.062029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:48.062029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:48.062117 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:48.062335 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:48.062339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:48.062469 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 09:04:48.062546 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 09:04:48.062663 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":134,"threads_started":1}
I20260928 09:04:48.063350 6432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33159
I20260928 09:04:48.063443 6432 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:48.063584 6432 heartbeater.cc:511] Master 127.0.98.62:33159 requested a full tablet report, sending...
I20260928 09:04:48.063933 5973 ts_manager.cc:194] Registered new tserver with Master: 5d16d8e68b4e41aaa886d0e1c2a7af10 (127.0.98.3:45471)
I20260928 09:04:48.064308 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46197
I20260928 09:04:48.070758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/test-admin.keytab.
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.087181 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 09:04:48.088917 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:48.088083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.0.98.1:44315)
0928 09:04:48.088211 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.088216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.088228 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:48.088282 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.088284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.088299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.088388 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.088810 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41874: BlockingRecv error: recv got EOF from 127.0.0.1:41874 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
W20260928 09:04:48.089061 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.090457 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.091409 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:48.099174 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:48.091993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.98.1:44315)
0928 09:04:48.092048 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.092050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.092067 (+ 17us) server_negotiation.cc:407] Connection header received
0928 09:04:48.092145 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.092145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.092156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.092207 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.092576 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.093258 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.094038 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.094361 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.094417 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:48.096436 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:48.096442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:48.096443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:48.096449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:48.098119 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.098428 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.098430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.098431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.098497 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.098662 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.098663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.098663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.098785 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:48.098890 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:48.098949 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 09:04:48.099015 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 09:04:48.101109 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:48.100480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.98.1:44315)
0928 09:04:48.100547 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.100549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.100633 (+ 84us) server_negotiation.cc:407] Connection header received
0928 09:04:48.100717 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.100718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.100730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.100785 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.101061 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41900: BlockingRecv error: recv got EOF from 127.0.0.1:41900 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 09:04:48.101182 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.102180 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.103039 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:48.109607 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:48.103572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.0.98.1:44315)
0928 09:04:48.103608 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.103614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.103622 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:48.103659 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.103660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.103666 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.103725 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.104081 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.104545 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.105191 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.105388 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.105463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:48.107500 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:48.107501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:48.107501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:48.107507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:48.108674 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.108968 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.108969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.108969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.109015 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.109174 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.109175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.109175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.109257 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:48.109341 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:48.109407 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 09:04:48.109485 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 09:04:48.111173 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:48.110611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.0.98.1:44315)
0928 09:04:48.110646 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.110648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.110717 (+ 69us) server_negotiation.cc:407] Connection header received
0928 09:04:48.110787 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.110788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.110798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.110847 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.111124 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41932: BlockingRecv error: recv got EOF from 127.0.0.1:41932 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 09:04:48.111399 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790586261566264-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.112341 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:48.113173 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:04:48.120052 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:48.113693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.98.1:44315)
0928 09:04:48.113732 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:48.113734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:48.113795 (+ 61us) server_negotiation.cc:407] Connection header received
0928 09:04:48.113866 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:48.113867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:48.113881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:48.113928 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:48.114327 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.114780 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.115498 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:48.115687 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:48.115796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:48.117749 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:48.117751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:48.117751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:48.117758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:48.118948 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.119304 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.119305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.119305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.119359 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:48.119575 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:48.119575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:48.119576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:48.119672 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:48.119791 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:48.119886 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 09:04:48.119959 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:04:48.120777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6035
I20260928 09:04:48.125865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6169
I20260928 09:04:48.130668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6303
I20260928 09:04:48.134985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 5941
2026-09-28T09:04:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2095 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[6452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[6452](info): set up 2 sockets
Sep 28 09:04:48 dist-test-slave-smns krb5kdc[6452](info): commencing operation
krb5kdc: starting...
W20260928 09:04:49.173954 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 09:04:49.298233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38561
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:49.380093 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:49.380316 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:49.380339 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:49.381507 6465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:49.381564 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:49.381580 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:49.381593 6465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:49.381605 6465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:49.383092 6465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:49.383289 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:49.383458 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:49.388607 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.388607 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.388674 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:49.388808 6465 server_base.cc:1034] running on GCE node
I20260928 09:04:49.389109 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:49.389456 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:49.390633 6465 hybrid_clock.cc:648] HybridClock initialized: now 1790586289390612 us; error 42 us; skew 500 ppm
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:49.393329 6465 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 09:04:49.393842 6465 webserver.cc:460] Webserver started at http://127.0.98.62:39553/ using document root <none> and password file <none>
I20260928 09:04:49.394106 6465 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:49.394181 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:49.394330 6465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:49.395385 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "635e5d7a743d46f2bc6bf623866b6bf2"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "92933a943c5b1ba0d274581c24882d5f"
server_key_iv: "be8f08e4d8135a3b468e68e80844d755"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.395752 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "635e5d7a743d46f2bc6bf623866b6bf2"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "92933a943c5b1ba0d274581c24882d5f"
server_key_iv: "be8f08e4d8135a3b468e68e80844d755"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.397083 6465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:49.397892 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.398077 6465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:49.398182 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "635e5d7a743d46f2bc6bf623866b6bf2"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "92933a943c5b1ba0d274581c24882d5f"
server_key_iv: "be8f08e4d8135a3b468e68e80844d755"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.398265 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:49.409951 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:49.412590 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:49.412770 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:49.417120 6465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38561
I20260928 09:04:49.417153 6532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38561 every 8 connection(s)
I20260928 09:04:49.417511 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:49.418632 6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:49.421053 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Bootstrap starting.
I20260928 09:04:49.421671 6533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:49.421917 6533 log.cc:824] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:49.422591 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: No bootstrap required, opened a new log
I20260928 09:04:49.423568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6465
I20260928 09:04:49.423689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:49.423903 392 external_mini_cluster.cc:1428] Setting key b8b910be1671318af85e72360ea20775
I20260928 09:04:49.425026 6533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } }
I20260928 09:04:49.425200 6533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:49.425263 6533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635e5d7a743d46f2bc6bf623866b6bf2, State: Initialized, Role: FOLLOWER
I20260928 09:04:49.425427 6533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } }
I20260928 09:04:49.425494 6533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:49.425516 6533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:49.425594 6533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:49.426558 6533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } }
I20260928 09:04:49.426687 6533 leader_election.cc:302] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635e5d7a743d46f2bc6bf623866b6bf2; no voters:
I20260928 09:04:49.427039 6533 leader_election.cc:288] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:49.427053 6538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:49.427197 6538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 635e5d7a743d46f2bc6bf623866b6bf2, State: Running, Role: LEADER
I20260928 09:04:49.427343 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } }
I20260928 09:04:49.427352 6533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:49.427841 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635e5d7a743d46f2bc6bf623866b6bf2. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } }
I20260928 09:04:49.427917 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:49.428158 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } }
I20260928 09:04:49.428290 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:49.428449 6543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:49.429157 6543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:49.431219 6543 catalog_manager.cc:1347] Generated new cluster ID: 33dd3e9fcb7f4bb98e93f1e8431b678f
I20260928 09:04:49.431270 6543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:49.436058 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.424456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.62:38561)
0928 09:04:49.424665 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:49.424673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:49.424699 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:04:49.425012 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:49.425020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:49.425069 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:49.425225 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:49.426116 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.427197 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.428267 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.428928 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.429040 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.432006 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:49.432010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:49.432011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:49.432023 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:49.434095 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.434520 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.434523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.434523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.434643 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.434880 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.434880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.434881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.435161 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:49.435654 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:49.435717 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 09:04:49.435861 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260928 09:04:49.477089 6543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:49.477900 6543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:49.482334 6543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Generated new TSK 0
I20260928 09:04:49.482566 6543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 09:04:49.514125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38561
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:49.600724 6559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:49.600987 6559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:49.601032 6559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:49.602389 6559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:49.602453 6559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:49.602499 6559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:49.604038 6559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:49.604318 6559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:49.604560 6559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:49.609791 6567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.609791 6565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.609795 6564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:49.609982 6559 server_base.cc:1034] running on GCE node
I20260928 09:04:49.610229 6559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:49.610503 6559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:49.611703 6559 hybrid_clock.cc:648] HybridClock initialized: now 1790586289611697 us; error 40 us; skew 500 ppm
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:49.619693 6559 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 09:04:49.620200 6559 webserver.cc:460] Webserver started at http://127.0.98.1:34971/ using document root <none> and password file <none>
I20260928 09:04:49.620450 6559 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:49.620523 6559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:49.620682 6559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:49.621855 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "842214b26a4245e7918c6e3051460681"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "c689cbdd843a30b89b08bf726d8037d8"
server_key_iv: "89b352179ecafc2b584403a13e439dfb"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.622249 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "842214b26a4245e7918c6e3051460681"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "c689cbdd843a30b89b08bf726d8037d8"
server_key_iv: "89b352179ecafc2b584403a13e439dfb"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.623627 6559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:49.624475 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.624691 6559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:49.624779 6559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "842214b26a4245e7918c6e3051460681"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "c689cbdd843a30b89b08bf726d8037d8"
server_key_iv: "89b352179ecafc2b584403a13e439dfb"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.624848 6559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:49.641325 6559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:49.643748 6559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:49.643906 6559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:49.644170 6559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:49.644496 6559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:49.644532 6559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.644554 6559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:49.644567 6559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.650295 6559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44273
I20260928 09:04:49.650384 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44273 every 8 connection(s)
I20260928 09:04:49.650699 6559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:49.659729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6559
I20260928 09:04:49.659843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:49.660152 392 external_mini_cluster.cc:1428] Setting key eca3e1f7ae101a92b122955847aa1df2
I20260928 09:04:49.660652 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.651692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57877 (local address 127.0.98.62:38561)
0928 09:04:49.651748 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:49.651750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:49.651761 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:49.652326 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:49.652327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:49.652340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:49.652379 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:49.653353 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.653945 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.655080 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.655331 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.655444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.657759 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:49.657761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:49.657761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:49.657769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:49.659101 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.659503 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.659504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.659504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.659555 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.659896 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.659897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.659897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.660031 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:49.660087 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:49.660443 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 09:04:49.660554 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 09:04:49.660801 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.651433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:57877)
0928 09:04:49.651635 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:49.651681 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:49.652210 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:49.652465 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:49.652468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:49.653051 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:49.653246 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:49.653250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.654059 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.654060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:49.654962 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:49.654965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.655449 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.655450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:49.655577 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.656128 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:49.656137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:49.657658 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:49.659179 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:49.659185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:49.659187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:49.659417 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:49.659640 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:49.659640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:49.659641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:49.659794 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:49.660100 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:49.660106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:49.660283 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 09:04:49.660429 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 09:04:49.660589 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260928 09:04:49.661286 6688 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38561
I20260928 09:04:49.661384 6688 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:49.661630 6688 heartbeater.cc:511] Master 127.0.98.62:38561 requested a full tablet report, sending...
I20260928 09:04:49.662181 6497 ts_manager.cc:194] Registered new tserver with Master: 842214b26a4245e7918c6e3051460681 (127.0.98.1:44273)
I20260928 09:04:49.663125 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57877
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 09:04:49.688302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38561
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:49.783794 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:49.784005 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:49.784088 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:49.785337 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:49.785392 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:49.785457 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:49.787645 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:49.787902 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:49.788154 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:49.793315 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.793423 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.793324 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:49.793615 6693 server_base.cc:1034] running on GCE node
I20260928 09:04:49.793869 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:49.794107 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:49.795248 6693 hybrid_clock.cc:648] HybridClock initialized: now 1790586289795223 us; error 41 us; skew 500 ppm
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:49.797814 6693 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 09:04:49.798262 6693 webserver.cc:460] Webserver started at http://127.0.98.2:37827/ using document root <none> and password file <none>
I20260928 09:04:49.798499 6693 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:49.798544 6693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:49.798708 6693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:49.799736 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea3200bcee5144a6be24f2f1ea138013"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "3f6680fe6a8f28256500a18694d7c7d2"
server_key_iv: "cd67af9247b4f6cb2145cbb598879f7b"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.800079 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea3200bcee5144a6be24f2f1ea138013"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "3f6680fe6a8f28256500a18694d7c7d2"
server_key_iv: "cd67af9247b4f6cb2145cbb598879f7b"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.801484 6693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:49.802268 6708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.802461 6693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:49.802531 6693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "ea3200bcee5144a6be24f2f1ea138013"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "3f6680fe6a8f28256500a18694d7c7d2"
server_key_iv: "cd67af9247b4f6cb2145cbb598879f7b"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.802620 6693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:49.815986 6693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:49.817898 6693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:49.818059 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:49.818282 6693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:49.818576 6693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:49.818609 6693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.818631 6693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:49.818676 6693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.824538 6693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42133
I20260928 09:04:49.824595 6821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42133 every 8 connection(s)
I20260928 09:04:49.824890 6693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:49.833712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6693
I20260928 09:04:49.833815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:49.833927 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.825662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60711 (local address 127.0.98.62:38561)
0928 09:04:49.825730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:49.825732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:49.825938 (+ 206us) server_negotiation.cc:407] Connection header received
0928 09:04:49.826399 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:49.826400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:49.826411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:49.826481 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:49.827439 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.828092 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.829227 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.829410 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.829488 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.831449 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:49.831450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:49.831451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:49.831456 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:49.832643 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.832990 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.832991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.832991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.833036 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.833245 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.833246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.833246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.833346 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:49.833417 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:49.833732 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 09:04:49.833824 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 09:04:49.834018 392 external_mini_cluster.cc:1428] Setting key 154caad440a5020f4f2a8bacbefdedf8
I20260928 09:04:49.834131 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.825658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:60711)
0928 09:04:49.825812 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:49.825853 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:49.826317 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:49.826535 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:49.826539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:49.827103 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:49.827286 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:49.827290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.828168 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.828169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:49.829156 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:49.829158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.829482 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.829482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:49.829649 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.830093 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:49.830101 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:49.831374 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:49.832713 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:49.832718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:49.832720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:49.832928 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:49.833092 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:49.833093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:49.833093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:49.833183 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:49.833421 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:49.833424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:49.833593 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 09:04:49.833717 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 09:04:49.833886 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260928 09:04:49.834582 6822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38561
I20260928 09:04:49.834719 6822 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:49.834921 6822 heartbeater.cc:511] Master 127.0.98.62:38561 requested a full tablet report, sending...
I20260928 09:04:49.835289 6497 ts_manager.cc:194] Registered new tserver with Master: ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133)
I20260928 09:04:49.835659 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60711
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 09:04:49.861056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38561
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:49.942162 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:49.942370 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:49.942394 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:49.943552 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:49.943595 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:49.943647 6827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:49.945122 6827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:49.945369 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:49.945655 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:49.950718 6835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:49.950695 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:49.950731 6827 server_base.cc:1034] running on GCE node
W20260928 09:04:49.950700 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:49.951122 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:49.951323 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:49.952477 6827 hybrid_clock.cc:648] HybridClock initialized: now 1790586289952468 us; error 30 us; skew 500 ppm
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:49.959272 6827 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 09:04:49.959748 6827 webserver.cc:460] Webserver started at http://127.0.98.3:34395/ using document root <none> and password file <none>
I20260928 09:04:49.960004 6827 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:49.960054 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:49.960225 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:49.961309 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "0571bcc2b0054c809593b81342c67d25"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "2507242fa3c5045d0a8bef353b9f144f"
server_key_iv: "9698ada4a16d33c4e7a149c2ce6c5574"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.961721 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0571bcc2b0054c809593b81342c67d25"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "2507242fa3c5045d0a8bef353b9f144f"
server_key_iv: "9698ada4a16d33c4e7a149c2ce6c5574"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.963047 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:49.963799 6842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.963995 6827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:49.964095 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "0571bcc2b0054c809593b81342c67d25"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "2507242fa3c5045d0a8bef353b9f144f"
server_key_iv: "9698ada4a16d33c4e7a149c2ce6c5574"
server_key_version: "encryptionkey@0"
I20260928 09:04:49.964166 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:49.976050 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:49.977898 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:49.978009 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:49.978299 6827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:49.978619 6827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:49.978674 6827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.978713 6827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:49.978740 6827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:49.984681 6827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35847
I20260928 09:04:49.984722 6955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35847 every 8 connection(s)
I20260928 09:04:49.985050 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:49.985342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6827
I20260928 09:04:49.985436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:49.985641 392 external_mini_cluster.cc:1428] Setting key 0f2d0e0589ef2e7720a1c51f11b53e65
Sep 28 09:04:49 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:49.994593 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.985828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50951 (local address 127.0.98.62:38561)
0928 09:04:49.985876 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:49.985879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:49.986178 (+ 299us) server_negotiation.cc:407] Connection header received
0928 09:04:49.986779 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:49.986780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:49.986791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:49.986831 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:49.987762 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.988267 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.989327 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.989584 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.989662 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.991748 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:49.991750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:49.991750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:49.991757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:49.993299 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.993746 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.993747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.993747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.993799 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:49.994077 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:49.994077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:49.994077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:49.994181 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:49.994227 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:49.994469 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 09:04:49.994528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:49.994855 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:49.985834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:50951)
0928 09:04:49.986048 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:49.986093 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:49.986674 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:49.986908 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:49.986912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:49.987480 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:49.987662 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:49.987667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.988356 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.988357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:49.989219 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:49.989222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:49.989680 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:49.989681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:49.989822 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:49.990284 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:49.990292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:49.991646 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:49.993379 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:49.993385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:49.993387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:49.993689 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:49.993930 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:49.993930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:49.993930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:49.994028 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:49.994232 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:49.994236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:49.994347 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 09:04:49.994453 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 09:04:49.994620 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260928 09:04:49.995333 6956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38561
I20260928 09:04:49.995414 6956 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:49.995595 6956 heartbeater.cc:511] Master 127.0.98.62:38561 requested a full tablet report, sending...
I20260928 09:04:49.996021 6497 ts_manager.cc:194] Registered new tserver with Master: 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847)
I20260928 09:04:49.996398 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50951
I20260928 09:04:49.997310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 09:04:50.024143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37361
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38561,127.0.98.61:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:50.107146 6961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:50.107394 6961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:50.107430 6961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:50.108691 6961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:50.108775 6961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:50.108803 6961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:50.108827 6961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:50.108889 6961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:50.110379 6961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38561,127.0.98.61:37361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
W20260928 09:04:50.110535 6961 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38561,127.0.98.61:37361'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 09:04:50.110695 6961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:50.110927 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:50.115866 6969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.115983 6961 server_base.cc:1034] running on GCE node
W20260928 09:04:50.115880 6966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.115866 6967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.116396 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.116693 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.117851 6961 hybrid_clock.cc:648] HybridClock initialized: now 1790586290117825 us; error 40 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.120503 6961 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 09:04:50.120970 6961 webserver.cc:460] Webserver started at http://127.0.98.61:42639/ using document root <none> and password file <none>
I20260928 09:04:50.121166 6961 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.121208 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.121299 6961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:50.122459 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/instance:
uuid: "98bce89f6168420d89392bff35393b3f"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "0b2b02ad7bd313aa656bab30fb3e3ec8"
server_key_iv: "739ec92c633366abbe4663fba8211303"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.122784 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal/instance:
uuid: "98bce89f6168420d89392bff35393b3f"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "0b2b02ad7bd313aa656bab30fb3e3ec8"
server_key_iv: "739ec92c633366abbe4663fba8211303"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.124081 6961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:04:50.124924 6976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.125123 6961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:50.125229 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
uuid: "98bce89f6168420d89392bff35393b3f"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "0b2b02ad7bd313aa656bab30fb3e3ec8"
server_key_iv: "739ec92c633366abbe4663fba8211303"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.125319 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.143400 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:50.145861 6961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:50.146034 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:50.150107 6961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37361
I20260928 09:04:50.150151 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37361 every 8 connection(s)
I20260928 09:04:50.150521 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:04:50.151585 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:50.153112 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:50.159504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 6961
I20260928 09:04:50.162483 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.153871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.98.62:38561)
0928 09:04:50.153935 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.153937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.154128 (+ 191us) server_negotiation.cc:407] Connection header received
0928 09:04:50.154564 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.154565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.154574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.154634 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.155228 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.155715 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.156709 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.156893 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.156948 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.159049 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.159051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.159052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.159060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.160920 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.161380 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.161382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.161382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.161453 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.161752 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.161753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.161753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.161944 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.162013 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.162301 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.162384 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:04:50.162663 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.153814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:41172)
0928 09:04:50.153981 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.154022 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.154477 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.154708 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.154711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.154952 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.155143 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.155146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.155798 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.155798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.156618 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.156620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.156967 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.156968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.157065 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.157499 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.157506 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.158940 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.161013 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.161019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.161021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.161298 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.161530 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.161531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.161531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.161681 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.162019 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.162024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.162146 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.162259 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.162434 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260928 09:04:50.163447 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 09:04:50.169505 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.160048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.0.98.61:37361)
0928 09:04:50.160258 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.160262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.160280 (+ 18us) server_negotiation.cc:407] Connection header received
0928 09:04:50.160326 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.160328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.160338 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.160398 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.161124 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.162134 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.163115 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.163515 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.163602 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.166223 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.166230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.166230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.166234 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.168212 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.168650 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.168653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.168653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.168719 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.168961 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.168961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.168961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.169099 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.169218 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.169340 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.169429 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260928 09:04:50.171468 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.163711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:56728)
0928 09:04:50.163762 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.163773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.163812 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.164062 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.164063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.164198 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.164235 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.164236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.164876 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.164877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.165672 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.165675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.166110 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.166111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.166190 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.166632 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.166637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.167862 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.170494 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.170495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.170496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.170654 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.170855 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.170856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.170856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.170907 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.171153 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.171154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.171190 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.171257 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.171353 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 09:04:50.171563 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.163763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.98.61:37361)
0928 09:04:50.163911 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.163913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.163919 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:04:50.163942 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.163942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.163950 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.164056 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.164309 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.164784 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.165772 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.166024 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.166085 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.167963 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.167964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.167964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.167971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.170409 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.170736 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.170737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.170737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.170793 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.170985 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.170986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.170986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.171089 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.171160 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.171281 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.171403 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 09:04:50.172792 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Bootstrap starting.
I20260928 09:04:50.173377 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:50.173695 7029 log.cc:824] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:50.174367 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: No bootstrap required, opened a new log
I20260928 09:04:50.175704 7029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } }
I20260928 09:04:50.175820 7029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:50.175841 7029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bce89f6168420d89392bff35393b3f, State: Initialized, Role: FOLLOWER
I20260928 09:04:50.175928 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } }
I20260928 09:04:50.176159 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:50.176127 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } }
I20260928 09:04:50.176237 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:04:50.177985 7047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:04:50.178043 7047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:04:50.179561 6961 master_runner.cc:428] Detected that this master 98bce89f6168420d89392bff35393b3f is joining an existing cluster
I20260928 09:04:50.179623 6961 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:37361
I20260928 09:04:50.187734 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.180639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.98.62:38561)
0928 09:04:50.180678 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.180681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.180850 (+ 169us) server_negotiation.cc:407] Connection header received
0928 09:04:50.180968 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.180969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.180980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.181063 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.181476 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.182073 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.182850 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.183060 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.183147 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.184194 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.184195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.184195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.184202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.186010 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.186950 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.186951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.186951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.187006 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.187211 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.187212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.187213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.187330 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.187406 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.187567 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.187641 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:50.187748 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.180597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:41184)
0928 09:04:50.180764 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.180774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.180879 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.181070 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.181070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.181328 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.181388 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.181389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.182159 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.182160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.182740 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.182741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.183147 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.183148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.183212 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.183702 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.183707 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.184125 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.186657 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.186657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.186657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.186874 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.187079 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.187080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.187081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.187139 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.187397 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.187398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.187440 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.187541 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.187641 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 09:04:50.188624 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.180822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:56740)
0928 09:04:50.181004 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.181012 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.181078 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.181196 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.181197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.181436 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.181496 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.181498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.182455 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.182456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.183301 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.183304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.184092 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.184093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.184161 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.184581 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.184585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.184958 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.186673 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.186674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.186674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.187011 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.187212 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.187213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.187213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.187765 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.188302 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.188303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.188347 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.188422 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.188521 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
I20260928 09:04:50.188743 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.180943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.61:37361)
0928 09:04:50.180977 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.180979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.181055 (+ 76us) server_negotiation.cc:407] Connection header received
0928 09:04:50.181133 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.181134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.181143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.181190 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.181618 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.182367 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.183694 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.184000 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.184082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.185064 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.185066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.185066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.185073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.186594 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.187097 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.187098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.187098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.187151 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.188084 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.188085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.188085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.188224 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.188300 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.188453 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.188596 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":18}
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 09:04:50.198436 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.191145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.0.98.61:37361)
0928 09:04:50.191199 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.191201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.191269 (+ 68us) server_negotiation.cc:407] Connection header received
0928 09:04:50.191366 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.191367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.191374 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.191421 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.191908 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.192372 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.193472 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.193706 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.193760 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.195963 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.195964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.195965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.195971 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.197299 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.197751 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.197752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.197752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.197799 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.198010 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.198011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.198011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.198097 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.198142 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.198324 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.198376 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:04:50.198529 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.191019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:56756)
0928 09:04:50.191184 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.191224 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.191294 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.191444 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.191448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.191781 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.191836 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.191838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.192448 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.192449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.193391 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.193394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.193773 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.193774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.193869 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.194293 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.194298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.195875 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.197382 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.197389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.197391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.197679 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.197864 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.197865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.197866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.197938 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.198156 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.198162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.198216 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.198308 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.198422 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260928 09:04:50.199203 6497 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:37361: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "98bce89f6168420d89392bff35393b3f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 37361
}
attrs {
promote: true
}
}
dest_uuid: "635e5d7a743d46f2bc6bf623866b6bf2"
cas_config_opid_index: -1
I20260928 09:04:50.199528 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } }
I20260928 09:04:50.200013 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 98bce89f6168420d89392bff35393b3f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.200331 7003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:50.201124 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:37361
I20260928 09:04:50.201211 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.201279 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:50.201282 7003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Refusing update from remote peer 635e5d7a743d46f2bc6bf623866b6bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 09:04:50.201628 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:50.201807 6961 master.cc:537] Master@127.0.98.61:37361 shutting down...
W20260928 09:04:50.203059 6484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 09:04:50.204402 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 -> Peer 98bce89f6168420d89392bff35393b3f (127.0.98.61:37361): Couldn't send request to peer 98bce89f6168420d89392bff35393b3f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 09:04:50.204748 6961 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:50.204840 6961 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:50.204890 6961 tablet_replica.cc:330] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: stopping tablet replica
I20260928 09:04:50.217433 6961 master.cc:552] Master@127.0.98.61:37361 shutdown complete.
I20260928 09:04:50.217481 6961 master_runner.cc:305] Clearing existing system tablet
I20260928 09:04:50.217655 6961 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.217720 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.218357 6961 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:50.218860 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.218986 6961 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.219043 6961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:50.219066 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
uuid: "98bce89f6168420d89392bff35393b3f"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "0b2b02ad7bd313aa656bab30fb3e3ec8"
server_key_iv: "739ec92c633366abbe4663fba8211303"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.219686 6961 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:50.221144 6961 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 09:04:50.221200 6961 log.cc:1196] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 09:04:50.221395 6961 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Deleting consensus metadata
I20260928 09:04:50.221517 6961 master_runner.cc:315] Copying system tablet from 127.0.98.62:38561
I20260928 09:04:50.222208 6961 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38561
I20260928 09:04:50.229031 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.222498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:41194)
0928 09:04:50.222612 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.222623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.222692 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.222842 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.222843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.222992 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.223043 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.223045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.224008 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.224009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.224563 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.224565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.225074 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.225074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.225150 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.225695 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.225700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.225996 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.227900 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.227901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.227901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.228087 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.228300 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.228300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.228300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.228352 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.228658 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.228659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.228698 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.228756 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.228879 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":77,"threads_started":1}
I20260928 09:04:50.229127 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.222587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.0.98.62:38561)
0928 09:04:50.222640 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.222642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.222691 (+ 49us) server_negotiation.cc:407] Connection header received
0928 09:04:50.222741 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.222742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.222753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.222839 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.223114 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.223915 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.224650 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.224993 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.225072 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.226096 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.226097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.226098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.226106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.227824 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.228169 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.228170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.228170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.228234 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.228415 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.228415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.228416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.228584 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.228652 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.228942 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.229027 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 09:04:50.229692 6517 tablet_copy_service.cc:140] P 635e5d7a743d46f2bc6bf623866b6bf2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 98bce89f6168420d89392bff35393b3f ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41194)
I20260928 09:04:50.229797 6517 tablet_copy_service.cc:161] P 635e5d7a743d46f2bc6bf623866b6bf2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 98bce89f6168420d89392bff35393b3f at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41194: session id = 98bce89f6168420d89392bff35393b3f-00000000000000000000000000000000
I20260928 09:04:50.230468 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 09:04:50.231098 6961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 09:04:50.232029 6961 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: tablet copy: Starting download of 0 data blocks...
I20260928 09:04:50.232198 6961 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: tablet copy: Starting download of 1 WAL segments...
I20260928 09:04:50.233175 6961 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 09:04:50.234040 6517 tablet_copy_service.cc:342] P 635e5d7a743d46f2bc6bf623866b6bf2: Request end of tablet copy session 98bce89f6168420d89392bff35393b3f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41194
I20260928 09:04:50.234113 6517 tablet_copy_service.cc:433] P 635e5d7a743d46f2bc6bf623866b6bf2: ending tablet copy session 98bce89f6168420d89392bff35393b3f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 98bce89f6168420d89392bff35393b3f
I20260928 09:04:50.236529 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:50.237792 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.237857 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.238023 6961 server_base.cc:1034] running on GCE node
W20260928 09:04:50.241729 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.241897 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.242049 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.243189 6961 hybrid_clock.cc:648] HybridClock initialized: now 1790586290243189 us; error 34 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.244546 6961 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 09:04:50.244837 6961 webserver.cc:460] Webserver started at http://127.0.98.61:34917/ using document root <none> and password file <none>
I20260928 09:04:50.244993 6961 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.245041 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.245714 6961 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.246253 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.246421 6961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.246475 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
uuid: "98bce89f6168420d89392bff35393b3f"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "0b2b02ad7bd313aa656bab30fb3e3ec8"
server_key_iv: "739ec92c633366abbe4663fba8211303"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.246536 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.252935 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:50.254781 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:50.257863 6961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37361
I20260928 09:04:50.257885 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37361 every 8 connection(s)
I20260928 09:04:50.258213 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:04:50.258862 7131 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:04:50.259280 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Bootstrap starting.
I20260928 09:04:50.261421 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:04:50.261680 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: Bootstrap complete.
I20260928 09:04:50.261884 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } }
I20260928 09:04:50.261952 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98bce89f6168420d89392bff35393b3f, State: Initialized, Role: LEARNER
I20260928 09:04:50.261979 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } }
I20260928 09:04:50.262180 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:50.262180 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.262300 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: LEARNER
I20260928 09:04:50.263628 7145 catalog_manager.cc:1259] Loaded cluster ID: 33dd3e9fcb7f4bb98e93f1e8431b678f
I20260928 09:04:50.263700 7145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: loading cluster ID for follower catalog manager: success
I20260928 09:04:50.264663 7145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: acquiring CA information for follower catalog manager: success
I20260928 09:04:50.264954 7145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:04:50.278657 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.271363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.0.98.61:37361)
0928 09:04:50.271517 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.271519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.271530 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:50.271556 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.271557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.271569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.271644 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.272172 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.273183 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.273949 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.274261 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.274345 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.275586 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.275588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.275588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.275595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.277263 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.277648 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.277650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.277650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.277711 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.277911 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.277912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.277912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.278053 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.278167 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.278297 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.278383 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 09:04:50.304623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35241
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:50.387845 7149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:50.388087 7149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:50.388109 7149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:50.389482 7149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:50.389532 7149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:50.389598 7149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:50.389621 7149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:50.389642 7149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:50.391330 7149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:50.391607 7149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:50.391860 7149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:50.396914 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.396986 7149 server_base.cc:1034] running on GCE node
W20260928 09:04:50.396904 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.396904 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.397503 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.397828 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.398998 7149 hybrid_clock.cc:648] HybridClock initialized: now 1790586290398986 us; error 30 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.401481 7149 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 09:04:50.401970 7149 webserver.cc:460] Webserver started at http://127.0.98.60:40681/ using document root <none> and password file <none>
I20260928 09:04:50.402153 7149 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.402194 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.402282 7149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:50.403290 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/instance:
uuid: "ec7ffeda80154ede9fa30499fea67144"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "e46267531bbbf42199eb4e44e8c8778a"
server_key_iv: "edc3e352877fbfdef933c927a9cf6dd5"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.403592 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal/instance:
uuid: "ec7ffeda80154ede9fa30499fea67144"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "e46267531bbbf42199eb4e44e8c8778a"
server_key_iv: "edc3e352877fbfdef933c927a9cf6dd5"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.405001 7149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:04:50.405815 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.406028 7149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:50.406128 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
uuid: "ec7ffeda80154ede9fa30499fea67144"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "e46267531bbbf42199eb4e44e8c8778a"
server_key_iv: "edc3e352877fbfdef933c927a9cf6dd5"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.406208 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.444365 7149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:50.447116 7149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:50.447306 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:50.451828 7149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35241
I20260928 09:04:50.451857 7216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35241 every 8 connection(s)
I20260928 09:04:50.452248 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:04:50.453331 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:50.454955 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:04:50.459894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 7149
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:50.465132 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.455769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.98.62:38561)
0928 09:04:50.455851 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.455854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.456012 (+ 158us) server_negotiation.cc:407] Connection header received
0928 09:04:50.456546 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.456548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.456560 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.456626 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.457235 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.457798 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.458803 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.459004 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.459105 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.461491 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.461493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.461493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.461500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.463463 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.464001 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.464003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.464003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.464060 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.464344 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.464345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.464345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.464503 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.464580 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.464938 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.465036 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:04:50.465288 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.455651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:41206)
0928 09:04:50.455875 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.455923 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.456409 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.456701 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.456704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.456953 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.457141 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.457144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.457890 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.457891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.458708 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.458710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.459122 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.459123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.459240 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.459707 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.459715 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.461371 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.463547 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.463553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.463555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.463925 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.464130 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.464131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.464131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.464273 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.464583 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.464590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.464734 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.464841 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.465035 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":114,"threads_started":1}
I20260928 09:04:50.466025 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 09:04:50.469920 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.460487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.0.98.60:35241)
0928 09:04:50.460628 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.460633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.460647 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:04:50.460664 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.460666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.460680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.460736 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.461437 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.462528 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.463682 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.464167 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.464234 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.467133 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.467140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.467140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.467147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.468674 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.469043 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.469047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.469047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.469107 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.469302 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.469303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.469303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.469507 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.469679 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.469785 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.469845 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 09:04:50.473445 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.466277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:56784)
0928 09:04:50.466323 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.466334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.466370 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.467002 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.467003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.467132 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.467173 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.467174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.468300 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.468301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.468778 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.468780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.469185 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.469185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.469252 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.469664 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.469670 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.471108 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.472483 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.472484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.472485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.472647 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.472853 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.472854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.472854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.472912 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.473156 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.473156 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.473196 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.473264 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.473356 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 09:04:50.473465 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.466795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.0.98.61:37361)
0928 09:04:50.466849 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.466851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.466858 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:50.466878 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.466878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.466889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.466987 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.467257 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.467737 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.468873 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.469101 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.469184 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.471196 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.471197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.471198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.471205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.472352 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.472718 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.472719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.472720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.472773 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.472979 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.472980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.472981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.473091 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.473165 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.473291 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.473369 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:50.473896 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 09:04:50.479905 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.474150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.0.1:58676)
0928 09:04:50.474197 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.474208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.474256 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.474373 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.474373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.474497 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.474520 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.474520 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.475125 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.475126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.475590 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.475591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.475983 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.475984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.476029 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.476346 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.476350 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.477517 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.478903 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.478904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.478904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.479095 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.479291 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.479292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.479292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.479351 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.479614 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.479615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.479649 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.479716 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.479804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 09:04:50.479964 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.474178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.0.98.60:35241)
0928 09:04:50.474204 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.474206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.474242 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:04:50.474308 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.474309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.474319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.474370 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.474590 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.475038 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.475679 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.475885 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.475984 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.477648 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.477649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.477650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.477656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.478799 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.479176 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.479177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.479178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.479230 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.479445 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.479445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.479446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.479547 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.479621 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.479743 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.479850 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":13}
I20260928 09:04:50.481165 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Bootstrap starting.
I20260928 09:04:50.482053 7217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:50.482337 7217 log.cc:824] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:50.483103 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: No bootstrap required, opened a new log
I20260928 09:04:50.484506 7217 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } }
I20260928 09:04:50.484633 7217 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:50.484656 7217 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec7ffeda80154ede9fa30499fea67144, State: Initialized, Role: FOLLOWER
I20260928 09:04:50.484795 7217 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } }
I20260928 09:04:50.485108 7217 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:50.485073 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } } }
I20260928 09:04:50.485177 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:04:50.486977 7233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:04:50.487038 7233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:04:50.489238 7149 master_runner.cc:428] Detected that this master ec7ffeda80154ede9fa30499fea67144 is joining an existing cluster
I20260928 09:04:50.489311 7149 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35241
I20260928 09:04:50.498337 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.490513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:56794)
0928 09:04:50.490693 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.490704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.490799 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.490970 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.490971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.491314 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.491400 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.491402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.492315 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.492316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.493708 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.493709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.494473 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.494473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.494517 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.494936 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.494940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.495653 (+ 713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.497384 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.497384 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.497385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.497592 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.497785 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.497786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.497786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.497837 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.498089 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.498089 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.498123 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.498173 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.498248 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":82,"thread_start_us":59,"threads_started":1}
I20260928 09:04:50.498368 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.490412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.0.98.61:37361)
0928 09:04:50.490466 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.490468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.490763 (+ 295us) server_negotiation.cc:407] Connection header received
0928 09:04:50.490869 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.490870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.490882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.490912 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.491491 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.492211 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.493788 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.494376 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.494411 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.495743 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.495744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.495745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.495752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.497301 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.497665 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.497666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.497666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.497724 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.497896 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.497897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.497897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.498026 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.498094 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.498196 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.498273 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:04:50.517002 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.490377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.98.62:38561)
0928 09:04:50.490424 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.490426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.490730 (+ 304us) server_negotiation.cc:407] Connection header received
0928 09:04:50.490832 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.490833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.490844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.490875 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.491376 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.492152 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.493368 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.493707 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.493752 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.513995 (+ 20243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.513998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.513998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.514008 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.515727 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516178 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.516179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.516180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.516242 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516456 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.516457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.516457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.516596 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.516664 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.516831 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.516904 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:50.517020 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.490333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:41214)
0928 09:04:50.490669 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.490678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.490756 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.490938 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.490939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.491208 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.491280 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.491282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.492256 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.492257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.493268 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.493271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.493845 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.493846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.493916 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.494358 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.494363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.513873 (+ 19510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.515820 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.515821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.515822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.516096 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.516317 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.516318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.516387 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.516671 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.516672 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.516724 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.516795 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.516903 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":66,"thread_start_us":79,"threads_started":1}
I20260928 09:04:50.517467 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.490600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.0.98.60:35241)
0928 09:04:50.490634 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.490636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.490769 (+ 133us) server_negotiation.cc:407] Connection header received
0928 09:04:50.490867 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.490868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.490876 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.490905 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.491483 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.492192 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.493655 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.493943 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.493978 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.514786 (+ 20808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.514787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.514788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.514795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.516431 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516756 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.516757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.516757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.516809 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516987 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.516987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.516988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.517094 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.517131 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.517292 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.517363 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260928 09:04:50.517747 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.490574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.0.1:58690)
0928 09:04:50.490722 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.490731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.490798 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.490960 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.490961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.491307 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.491402 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.491403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.492285 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.492286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.493569 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.493571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.494041 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.494041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.494095 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.514317 (+ 20222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.514323 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.514674 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.516497 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.516498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.516690 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.516876 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.516877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.516877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.516925 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.517168 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.517169 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.517200 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.517566 (+ 366us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.517652 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":18,"thread_start_us":78,"threads_started":1}
I20260928 09:04:50.521334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6559
I20260928 09:04:50.522179 7222 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:50.522293 7222 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } }
I20260928 09:04:50.522578 7222 leader_election.cc:288] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561), 98bce89f6168420d89392bff35393b3f (127.0.98.61:37361)
I20260928 09:04:50.522802 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec7ffeda80154ede9fa30499fea67144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bce89f6168420d89392bff35393b3f" is_pre_election: true
I20260928 09:04:50.522912 7105 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Handling vote request from an unknown peer ec7ffeda80154ede9fa30499fea67144
I20260928 09:04:50.522969 7105 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate ec7ffeda80154ede9fa30499fea67144 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586289, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 09:04:50.527431 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec7ffeda80154ede9fa30499fea67144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" is_pre_election: true
I20260928 09:04:50.527714 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.519518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.0.98.60:35241)
0928 09:04:50.519551 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.519553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.519564 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:50.519605 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.519606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.519613 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.519656 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.519936 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.520372 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.521362 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.521724 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.521785 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.524232 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.524233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.524234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.524241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.526138 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.526542 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.526543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.526544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.526595 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.526798 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.526799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.526800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.526935 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.526992 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.527500 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.527579 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:04:50.527447 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.519464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.0.1:58706)
0928 09:04:50.519509 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.519519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.519553 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.519668 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.519669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.519827 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.519872 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.519874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.520454 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.520455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.521271 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.521274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.522022 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.522023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.522093 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.522637 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.522643 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.524139 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.526227 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.526228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.526228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.526467 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.526658 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.526659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.526659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.526719 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.527017 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.527018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.527066 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.527166 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.527279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 09:04:50.529016 7165 leader_election.cc:302] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec7ffeda80154ede9fa30499fea67144; no voters: 635e5d7a743d46f2bc6bf623866b6bf2, 98bce89f6168420d89392bff35393b3f
I20260928 09:04:50.529155 7222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:50.529713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44273
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38561
--webserver_port=34971
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:04:50.530206 7222 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 09:04:50.530879 6497 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35241: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ec7ffeda80154ede9fa30499fea67144"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35241
}
attrs {
promote: true
}
}
dest_uuid: "635e5d7a743d46f2bc6bf623866b6bf2"
cas_config_opid_index: 5
I20260928 09:04:50.531157 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } }
I20260928 09:04:50.531546 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ec7ffeda80154ede9fa30499fea67144 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.531828 7191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Refusing update from remote peer 635e5d7a743d46f2bc6bf623866b6bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 09:04:50.532570 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35241
I20260928 09:04:50.532626 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:50.532843 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.532902 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:50.534103 7191 raft_consensus.cc:858] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } }
W20260928 09:04:50.534168 7191 raft_consensus.cc:1565] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } }
I20260928 09:04:50.534706 7191 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 98bce89f6168420d89392bff35393b3f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.536322 7149 master.cc:537] Master@127.0.98.60:35241 shutting down...
I20260928 09:04:50.536661 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635e5d7a743d46f2bc6bf623866b6bf2. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.536748 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 09:04:50.536785 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.536839 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 09:04:50.536871 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } }
I20260928 09:04:50.536916 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 09:04:50.537871 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.537967 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
W20260928 09:04:50.538475 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 -> Peer ec7ffeda80154ede9fa30499fea67144 (127.0.98.60:35241): Couldn't send request to peer ec7ffeda80154ede9fa30499fea67144. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 09:04:50.539127 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.533263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:56798)
0928 09:04:50.533605 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.533616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.533665 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.533936 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.533937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.533939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:50.533984 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.533985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.535073 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.535074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.536288 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.536291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.538787 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.538789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.538877 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.538911 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.538976 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.539030 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260928 09:04:50.539431 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.533358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.0.98.61:37361)
0928 09:04:50.533538 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.533576 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.533718 (+ 142us) server_negotiation.cc:407] Connection header received
0928 09:04:50.533743 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.533744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.533766 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.533848 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:50.534061 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.534835 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.537824 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.538712 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.538791 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.538874 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.539276 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.539339 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 09:04:50.539636 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Refusing update from remote peer 635e5d7a743d46f2bc6bf623866b6bf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 09:04:50.539834 7149 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 09:04:50.539916 7149 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 09:04:50.539948 7149 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: stopping tablet replica
I20260928 09:04:50.540153 7243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 09:04:50.540788 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ec7ffeda80154ede9fa30499fea67144 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.542212 7243 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: attempting to promote NON_VOTER 98bce89f6168420d89392bff35393b3f to VOTER
I20260928 09:04:50.542279 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635e5d7a743d46f2bc6bf623866b6bf2. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.542389 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: LEARNER
I20260928 09:04:50.542436 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: true } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.542408 7243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } }
I20260928 09:04:50.542478 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: LEARNER
I20260928 09:04:50.543030 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 LEARNER]: Refusing update from remote peer 635e5d7a743d46f2bc6bf623866b6bf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 09:04:50.543097 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 -> Peer ec7ffeda80154ede9fa30499fea67144 (127.0.98.60:35241): Couldn't send request to peer ec7ffeda80154ede9fa30499fea67144. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 09:04:50.543263 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 09:04:50.543947 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 98bce89f6168420d89392bff35393b3f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.544173 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 98bce89f6168420d89392bff35393b3f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.544790 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.544857 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:50.544904 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.544941 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:50.545607 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.545742 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:04:50.558334 7149 master.cc:552] Master@127.0.98.60:35241 shutdown complete.
I20260928 09:04:50.558386 7149 master_runner.cc:305] Clearing existing system tablet
I20260928 09:04:50.558512 7149 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.558548 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.559268 7149 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:50.559787 7254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.559926 7149 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.559995 7149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.560021 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
uuid: "ec7ffeda80154ede9fa30499fea67144"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "e46267531bbbf42199eb4e44e8c8778a"
server_key_iv: "edc3e352877fbfdef933c927a9cf6dd5"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.560709 7149 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:50.562330 7149 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 09:04:50.562398 7149 log.cc:1196] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 09:04:50.562645 7149 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Deleting consensus metadata
I20260928 09:04:50.562772 7149 master_runner.cc:315] Copying system tablet from 127.0.98.62:38561
I20260928 09:04:50.563366 7149 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38561
I20260928 09:04:50.570773 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.563669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:41226)
0928 09:04:50.563819 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.563830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.563882 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.564045 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.564046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.564234 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.564285 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.564287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.565268 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.565269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.566196 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.566199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.566676 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.566677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.566733 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.567284 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.567290 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.567681 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.569600 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.569601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.569602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.569823 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.570049 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.570049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.570050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.570114 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.570398 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.570399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.570455 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.570565 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.570673 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260928 09:04:50.570788 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.563811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.0.98.62:38561)
0928 09:04:50.563859 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.563861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.563873 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:50.563943 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.563944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.563955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.564038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.564388 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.565166 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.566293 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.566588 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.566671 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.567774 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.567775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.567776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.567783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.569478 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.569907 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.569908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.569909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.569973 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.570184 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.570185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.570185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.570326 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.570397 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.570613 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.570693 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:04:50.571523 6517 tablet_copy_service.cc:140] P 635e5d7a743d46f2bc6bf623866b6bf2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ec7ffeda80154ede9fa30499fea67144 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41226)
I20260928 09:04:50.571583 6517 tablet_copy_service.cc:161] P 635e5d7a743d46f2bc6bf623866b6bf2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ec7ffeda80154ede9fa30499fea67144 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41226: session id = ec7ffeda80154ede9fa30499fea67144-00000000000000000000000000000000
I20260928 09:04:50.572134 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 09:04:50.572638 7149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 09:04:50.574025 7149 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: tablet copy: Starting download of 0 data blocks...
I20260928 09:04:50.574174 7149 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: tablet copy: Starting download of 1 WAL segments...
I20260928 09:04:50.574932 7149 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 09:04:50.575831 6517 tablet_copy_service.cc:342] P 635e5d7a743d46f2bc6bf623866b6bf2: Request end of tablet copy session ec7ffeda80154ede9fa30499fea67144-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41226
I20260928 09:04:50.575878 6517 tablet_copy_service.cc:433] P 635e5d7a743d46f2bc6bf623866b6bf2: ending tablet copy session ec7ffeda80154ede9fa30499fea67144-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ec7ffeda80154ede9fa30499fea67144
I20260928 09:04:50.578583 7149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:50.579841 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.582451 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.582638 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.582808 7149 server_base.cc:1034] running on GCE node
I20260928 09:04:50.582921 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.583062 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.584184 7149 hybrid_clock.cc:648] HybridClock initialized: now 1790586290584186 us; error 61 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.585736 7149 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 09:04:50.585994 7149 webserver.cc:460] Webserver started at http://127.0.98.60:42453/ using document root <none> and password file <none>
I20260928 09:04:50.586117 7149 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.586156 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.586925 7149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:50.587522 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.587693 7149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:50.587749 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
uuid: "ec7ffeda80154ede9fa30499fea67144"
format_stamp: "Formatted at 2026-09-28 09:04:50 on dist-test-slave-smns"
server_key: "e46267531bbbf42199eb4e44e8c8778a"
server_key_iv: "edc3e352877fbfdef933c927a9cf6dd5"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.587808 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.627550 7149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260928 09:04:50.627873 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:50.628198 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:50.628247 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 09:04:50.629628 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260928 09:04:50.630067 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:50.630125 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:50.630177 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:50.632459 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34971
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:50.632732 7149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35241
I20260928 09:04:50.632735 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:50.632980 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35241 every 8 connection(s)
I20260928 09:04:50.633013 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:04:50.633054 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:04:50.634084 7327 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:04:50.634652 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Bootstrap starting.
I20260928 09:04:50.637367 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:04:50.637603 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Bootstrap complete.
I20260928 09:04:50.637820 7327 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } }
I20260928 09:04:50.637913 7327 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec7ffeda80154ede9fa30499fea67144, State: Initialized, Role: LEARNER
I20260928 09:04:50.637950 7327 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } }
I20260928 09:04:50.638118 7327 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:50.638120 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:50.638348 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: LEARNER
W20260928 09:04:50.639897 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.640098 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.640347 7242 server_base.cc:1034] running on GCE node
I20260928 09:04:50.640906 7347 catalog_manager.cc:1259] Loaded cluster ID: 33dd3e9fcb7f4bb98e93f1e8431b678f
I20260928 09:04:50.640969 7347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: loading cluster ID for follower catalog manager: success
I20260928 09:04:50.641755 7347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: acquiring CA information for follower catalog manager: success
I20260928 09:04:50.642078 7347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 09:04:50.642444 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.642689 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.643034 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.644208 7242 hybrid_clock.cc:648] HybridClock initialized: now 1790586290644184 us; error 38 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.647181 7242 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 09:04:50.647677 7242 webserver.cc:460] Webserver started at http://127.0.98.1:34971/ using document root <none> and password file <none>
I20260928 09:04:50.647909 7242 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.647975 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.650171 7242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:50.650821 7353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.651038 7242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:50.651125 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "842214b26a4245e7918c6e3051460681"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "c689cbdd843a30b89b08bf726d8037d8"
server_key_iv: "89b352179ecafc2b584403a13e439dfb"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.651553 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.665306 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:50.667781 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:50.667904 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:50.668088 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:50.668396 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:50.668437 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.668460 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:50.668474 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.674790 7242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44273
I20260928 09:04:50.674839 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44273 every 8 connection(s)
I20260928 09:04:50.675190 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:50.675531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 7242
I20260928 09:04:50.675693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6693
I20260928 09:04:50.682252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42133
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38561
--webserver_port=37827
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 09:04:50.689677 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.676271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39449 (local address 127.0.98.61:37361)
0928 09:04:50.676321 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.676323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.676580 (+ 257us) server_negotiation.cc:407] Connection header received
0928 09:04:50.677414 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.677422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.677434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.677509 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.679021 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.679861 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.681640 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.681964 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.682010 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.685865 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.685866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.685867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.685875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.687718 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688480 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.688481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.688482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.688543 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688842 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.688843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.688843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.689003 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.689060 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.689468 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.689573 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:50.689754 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.676528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49849 (local address 127.0.98.62:38561)
0928 09:04:50.676575 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.676577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.678041 (+ 1464us) server_negotiation.cc:407] Connection header received
0928 09:04:50.678147 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.678149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.678161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.678192 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.679021 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.679817 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.681612 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.681914 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.681962 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.685266 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.685267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.685268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.685277 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.688121 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688683 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.688685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.688685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.688752 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688989 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.688990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.688991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.689151 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.689201 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.689475 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.689615 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:50.690133 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.675987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.98.1:39449)
0928 09:04:50.676424 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.676488 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.677291 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.677613 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.677618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.678594 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.678894 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.678901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.679970 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.679971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.681475 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.681489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.682200 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.682201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.682267 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.683647 (+ 1380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.683654 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.685775 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.687911 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.687914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.687924 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.688381 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.688622 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.688623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.688763 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.689091 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.689097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.689275 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.689818 (+ 543us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.689960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":123,"thread_start_us":90,"threads_started":1}
I20260928 09:04:50.690222 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.676301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:49849)
0928 09:04:50.677982 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.677990 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.678067 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.678256 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.678257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.678578 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.678894 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.678901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.679957 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.679958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.681474 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.681476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.682060 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.682061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.682226 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.682918 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.682930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.685146 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.688274 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.688275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.688581 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.688833 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.688834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.688901 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.689255 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.689255 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.689316 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.689881 (+ 565us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.690034 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"mutex_wait_us":169,"thread_start_us":68,"threads_started":1}
I20260928 09:04:50.690696 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37361
I20260928 09:04:50.690810 7469 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.690858 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.676495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41883 (local address 127.0.98.60:35241)
0928 09:04:50.676650 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.676652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.678022 (+ 1370us) server_negotiation.cc:407] Connection header received
0928 09:04:50.678055 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.678056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.678070 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.678141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.679017 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.680054 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.681608 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.681928 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.681965 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.685913 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.685914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.685924 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.685932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.687832 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688516 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.688516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.688517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.688594 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688833 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.688834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.688834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.689002 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.689061 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.690635 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.690736 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":52,"threads_started":1}
I20260928 09:04:50.691035 7469 heartbeater.cc:511] Master 127.0.98.61:37361 requested a full tablet report, sending...
I20260928 09:04:50.691265 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38561
I20260928 09:04:50.691309 7467 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.691337 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.675997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.98.1:41883)
0928 09:04:50.677804 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.677817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.677879 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.678192 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.678192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.678578 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.678894 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.678901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.680154 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.680154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.681473 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.681476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.682061 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.682062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.682226 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.683018 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.683022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.685517 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.687905 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.687914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.687917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.688437 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.688664 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.688665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.688665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.688763 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.689090 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.689097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.689275 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.689431 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.689642 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1785,"mutex_wait_us":140,"spinlock_wait_cycles":7552,"thread_start_us":97,"threads_started":1}
I20260928 09:04:50.691378 7467 heartbeater.cc:511] Master 127.0.98.62:38561 requested a full tablet report, sending...
I20260928 09:04:50.691653 7095 ts_manager.cc:194] Registered new tserver with Master: 842214b26a4245e7918c6e3051460681 (127.0.98.1:44273)
I20260928 09:04:50.691715 6497 ts_manager.cc:194] Re-registered known tserver with Master: 842214b26a4245e7918c6e3051460681 (127.0.98.1:44273)
I20260928 09:04:50.692282 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49849
I20260928 09:04:50.692589 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35241
I20260928 09:04:50.692647 7468 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.692728 7468 heartbeater.cc:511] Master 127.0.98.60:35241 requested a full tablet report, sending...
I20260928 09:04:50.693094 7291 ts_manager.cc:194] Registered new tserver with Master: 842214b26a4245e7918c6e3051460681 (127.0.98.1:44273)
W20260928 09:04:50.769354 7475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:50.769634 7475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:50.769704 7475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:50.770918 7475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:50.770964 7475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:50.770998 7475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:50.772532 7475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37827
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:50.772804 7475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:50.773052 7475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:50.778177 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.778194 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.778168 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.778484 7475 server_base.cc:1034] running on GCE node
I20260928 09:04:50.778748 7475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.779011 7475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.780201 7475 hybrid_clock.cc:648] HybridClock initialized: now 1790586290780196 us; error 38 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.782819 7475 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 09:04:50.783245 7475 webserver.cc:460] Webserver started at http://127.0.98.2:37827/ using document root <none> and password file <none>
I20260928 09:04:50.783489 7475 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.783578 7475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.784981 7475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:50.785579 7490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.785743 7475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:50.785859 7475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "ea3200bcee5144a6be24f2f1ea138013"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "3f6680fe6a8f28256500a18694d7c7d2"
server_key_iv: "cd67af9247b4f6cb2145cbb598879f7b"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.786230 7475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.825455 7475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:50.828296 7475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:50.828420 7475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:50.828609 7475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:50.828920 7475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:50.828975 7475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.829032 7475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:50.829077 7475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.835063 7475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42133
I20260928 09:04:50.835142 7603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42133 every 8 connection(s)
I20260928 09:04:50.835600 7475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:50.841821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 7475
I20260928 09:04:50.842007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6827
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 09:04:50.849133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35847
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38561
--webserver_port=34395
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:04:50.852773 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.837251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56911 (local address 127.0.98.60:35241)
0928 09:04:50.837341 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.837344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.837353 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:50.837737 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.837738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.837753 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.837807 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.838763 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.839547 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841198 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.841508 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841572 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.849261 (+ 7689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.849263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.849264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.849271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.850914 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851613 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.851614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.851615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.851667 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.852008 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.852009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.852009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.852142 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.852186 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.852592 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.852682 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 09:04:50.852828 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.836929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56715 (local address 127.0.98.62:38561)
0928 09:04:50.836969 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.836971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.837045 (+ 74us) server_negotiation.cc:407] Connection header received
0928 09:04:50.837606 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.837608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.837624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.837659 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.838765 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.839570 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841198 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.841508 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841572 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.848598 (+ 7026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.848599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.848600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.848610 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.851205 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851673 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.851674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.851674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.851732 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.852000 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.852001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.852001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.852150 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.852191 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.852592 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.852684 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:50.853232 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.836513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53019 (local address 127.0.98.61:37361)
0928 09:04:50.836592 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.836595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.837175 (+ 580us) server_negotiation.cc:407] Connection header received
0928 09:04:50.837590 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.837591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.837605 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.837679 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.838809 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.839711 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841220 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.841661 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841748 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.848345 (+ 6597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.848347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.848347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.848356 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.850247 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851606 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.851607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.851608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.851667 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.852001 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.852002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.852002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.852142 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.852192 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.853061 (+ 869us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.853144 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:04:50.853237 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.836449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.98.2:56911)
0928 09:04:50.836618 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.836660 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.837385 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.837892 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.837896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.838365 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.838650 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.838656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.839686 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.839687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.841090 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.841093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.841668 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.841818 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.845854 (+ 4036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.845864 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.849136 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.851024 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.851038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.851516 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.851741 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.851742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.851922 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.852233 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.852234 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.852429 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.852910 (+ 481us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.853022 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":75,"thread_start_us":82,"threads_started":1}
I20260928 09:04:50.853618 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.836449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.98.2:53019)
0928 09:04:50.837115 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.837125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.837399 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.837914 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.837915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.838445 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.838651 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.838658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.839798 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.839798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.841117 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.841120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.841838 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.842005 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.843104 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.843116 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.848232 (+ 5116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.851036 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.851039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.851516 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.851740 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.851741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.851927 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.852226 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.852234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.852423 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.852550 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.852723 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":787,"thread_start_us":71,"threads_started":1}
I20260928 09:04:50.853801 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35241
I20260928 09:04:50.853914 7605 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.854079 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37361
I20260928 09:04:50.854133 7606 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.854167 7605 heartbeater.cc:511] Master 127.0.98.60:35241 requested a full tablet report, sending...
I20260928 09:04:50.854210 7606 heartbeater.cc:511] Master 127.0.98.61:37361 requested a full tablet report, sending...
I20260928 09:04:50.854539 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.836762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:56715)
0928 09:04:50.836905 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.836944 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.837476 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.837892 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.837896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.838419 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.838650 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.838656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.839698 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.839698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.841090 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.841093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.841667 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.841668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.841818 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.843741 (+ 1923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.843747 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.848496 (+ 4749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.851306 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.851308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.851581 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.851805 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.851806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.851806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.851926 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.852234 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.852235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.852424 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.853257 (+ 833us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.853649 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":58,"thread_start_us":76,"threads_started":1}
I20260928 09:04:50.854694 7095 ts_manager.cc:194] Registered new tserver with Master: ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133)
I20260928 09:04:50.854694 7291 ts_manager.cc:194] Registered new tserver with Master: ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133)
I20260928 09:04:50.855111 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38561
I20260928 09:04:50.855180 7604 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.855252 7604 heartbeater.cc:511] Master 127.0.98.62:38561 requested a full tablet report, sending...
I20260928 09:04:50.855518 6497 ts_manager.cc:194] Re-registered known tserver with Master: ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133)
I20260928 09:04:50.856096 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56715
W20260928 09:04:50.934643 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:50.934875 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:50.934911 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:50.936051 7611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:50.936087 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:50.936113 7611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:50.937534 7611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34395
--tserver_master_addrs=127.0.98.62:38561,127.0.98.61:37361,127.0.98.60:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:50.937898 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:50.938252 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:50.943625 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.943670 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:50.943647 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:50.943634 7611 server_base.cc:1034] running on GCE node
I20260928 09:04:50.944015 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:50.944262 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:50.945441 7611 hybrid_clock.cc:648] HybridClock initialized: now 1790586290945417 us; error 34 us; skew 500 ppm
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:50.948006 7611 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 09:04:50.948393 7611 webserver.cc:460] Webserver started at http://127.0.98.3:34395/ using document root <none> and password file <none>
I20260928 09:04:50.948580 7611 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:50.948630 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:50.949975 7611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:50.950527 7626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.950728 7611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:50.950817 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "0571bcc2b0054c809593b81342c67d25"
format_stamp: "Formatted at 2026-09-28 09:04:49 on dist-test-slave-smns"
server_key: "2507242fa3c5045d0a8bef353b9f144f"
server_key_iv: "9698ada4a16d33c4e7a149c2ce6c5574"
server_key_version: "encryptionkey@0"
I20260928 09:04:50.951191 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:50.967088 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:50.969208 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:50.969327 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:50.969522 7611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:50.969880 7611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:50.969914 7611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.969936 7611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:50.969949 7611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:50.976056 7611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35847
I20260928 09:04:50.976130 7739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35847 every 8 connection(s)
I20260928 09:04:50.976475 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 09:04:50.985077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 7611
Sep 28 09:04:50 dist-test-slave-smns krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 09:04:50.988715 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.977536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36935 (local address 127.0.98.61:37361)
0928 09:04:50.977606 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.977608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.977620 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:04:50.978637 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.978638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.978652 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.978708 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.979499 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.979992 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981449 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.981779 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.985172 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.985174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.985174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.985182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.987089 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.987613 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.987614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.987615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.987676 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.987988 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.987988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.987989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.988112 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.988193 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.988541 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.988621 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:04:50.989876 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.977638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43971 (local address 127.0.98.60:35241)
0928 09:04:50.977682 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.977685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.977792 (+ 107us) server_negotiation.cc:407] Connection header received
0928 09:04:50.978645 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.978647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.978659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.978693 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.979495 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.980258 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.982320 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.982657 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.982727 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.985791 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.985792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.985793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.985800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.987717 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.989065 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.989066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.989067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.989127 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.989319 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.989320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.989321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.989432 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.989482 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.989684 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.989769 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:04:50.990093 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.977386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.98.3:43971)
0928 09:04:50.977740 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.977748 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.978546 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.978789 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.978792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.979187 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.979385 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.979391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.981452 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.982223 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.982225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.982733 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.982734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.982824 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.983226 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.983230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.985679 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.987800 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.987801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.987801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.988037 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.989198 (+ 1161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.989199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.989200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.989259 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.989506 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.989507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.989565 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.989632 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.989819 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260928 09:04:50.991722 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.977501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58535 (local address 127.0.98.62:38561)
0928 09:04:50.977589 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.977591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.977701 (+ 110us) server_negotiation.cc:407] Connection header received
0928 09:04:50.978646 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.978647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.978671 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.978701 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.979506 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.980268 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981449 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.981779 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.984688 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.984689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.984689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.984697 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.988954 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.990904 (+ 1950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.990904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.990905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.990961 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.991143 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.991143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.991144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.991267 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.991321 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.991555 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.991630 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 09:04:50.991739 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.977201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:58535)
0928 09:04:50.977627 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.977637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.978541 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.978792 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.978792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.979207 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.979384 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.979391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.980339 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.980340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.981346 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.981348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.981924 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.982008 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.982994 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.982999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.984590 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.990599 (+ 6009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.990600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.990600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.990824 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.991026 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.991026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.991027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.991077 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.991373 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.991374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.991431 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.991523 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.991627 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":10,"thread_start_us":100,"threads_started":1}
I20260928 09:04:50.991818 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35241
I20260928 09:04:50.991927 7741 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.992128 7741 heartbeater.cc:511] Master 127.0.98.60:35241 requested a full tablet report, sending...
I20260928 09:04:50.992161 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38561
I20260928 09:04:50.992244 7740 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.992390 7740 heartbeater.cc:511] Master 127.0.98.62:38561 requested a full tablet report, sending...
I20260928 09:04:50.992547 7291 ts_manager.cc:194] Registered new tserver with Master: 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847)
I20260928 09:04:50.992758 6497 ts_manager.cc:194] Re-registered known tserver with Master: 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847)
I20260928 09:04:50.993260 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58535
I20260928 09:04:50.994668 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.977201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.98.3:36935)
0928 09:04:50.977380 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:50.977432 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:50.978520 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:50.978787 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:50.978791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:50.979190 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:50.979384 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.979389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.980099 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.980100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.981346 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:50.981348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.981871 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.981872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:50.982011 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.982687 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:50.982716 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:50.985057 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:50.987173 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.987181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.987183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.987519 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.987756 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:50.987756 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:50.987756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:50.987924 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:50.988184 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:50.988191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:50.988360 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 09:04:50.994055 (+ 5695us) client_negotiation.cc:240] Negotiation successful
0928 09:04:50.994173 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":6,"thread_start_us":112,"threads_started":1}
I20260928 09:04:50.995013 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37361
I20260928 09:04:50.995066 7742 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:50.995139 7742 heartbeater.cc:511] Master 127.0.98.61:37361 requested a full tablet report, sending...
I20260928 09:04:50.995438 7095 ts_manager.cc:194] Registered new tserver with Master: 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847)
I20260928 09:04:50.996410 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.987216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.0.98.62:38561)
0928 09:04:50.987465 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.987467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.987642 (+ 175us) server_negotiation.cc:407] Connection header received
0928 09:04:50.987666 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.987667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.987680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.987738 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.988448 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.989307 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.990837 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.991158 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.991215 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.992927 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.992928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.992929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.992935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.995190 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.995591 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.995592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.995593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.995654 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.995856 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.995857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.995857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.995972 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.996082 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.996190 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.996273 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
I20260928 09:04:50.998852 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.986741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.0.98.60:35241)
0928 09:04:50.986886 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.986887 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.987070 (+ 183us) server_negotiation.cc:407] Connection header received
0928 09:04:50.987169 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.987169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.987182 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.987222 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.988611 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.989702 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.993037 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.993395 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.993587 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.995361 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.995362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.995362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.995369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.997325 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.997790 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.997791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.997791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.997855 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.998215 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.998216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.998216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.998361 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.998481 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:50.998578 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 09:04:50.998670 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260928 09:04:50.998566 6497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:51.000249 6497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:51.001243 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:50.986384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.0.98.61:37361)
0928 09:04:50.988827 (+ 2443us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:50.988828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:50.988835 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:50.988852 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:50.988853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:50.988861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:50.989759 (+ 898us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:50.989789 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.990502 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.993011 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:50.993302 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:50.993358 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:50.994857 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:50.994858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:50.994859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:50.994866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:50.996948 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.997757 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.997758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.997758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.997826 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:50.998040 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:50.998041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:50.998042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:50.998182 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:50.998267 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.001045 (+ 2778us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.001146 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2430,"thread_start_us":54,"threads_started":1}
I20260928 09:04:51.008070 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.004336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35847 (local address 127.0.0.1:41138)
0928 09:04:51.004372 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:51.004383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:51.004425 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:51.004767 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:51.004768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:51.004770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:51.004827 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.004830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.006043 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.006044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.006927 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.006930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.007831 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.007832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.007892 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.007896 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:51.007958 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:04:51.007998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 09:04:51.008096 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.004411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.0.98.3:35847)
0928 09:04:51.004583 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.004589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.004600 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:51.004641 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.004643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.004660 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.004905 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:51.004961 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.005953 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.007017 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.007736 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.007819 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.007915 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.007974 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.008010 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260928 09:04:51.008590 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.004573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42133 (local address 127.0.0.1:51694)
0928 09:04:51.004737 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:51.004747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:51.004790 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:51.005339 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:51.005339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:51.005341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:51.005367 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.005367 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.006588 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.006589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.007472 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.007474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.008344 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.008345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.008401 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.008406 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:51.008455 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 09:04:51.008490 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":35,"thread_start_us":64,"threads_started":1}
I20260928 09:04:51.008986 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.005007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.0.98.2:42133)
0928 09:04:51.005202 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.005206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.005215 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:51.005253 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.005255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.005271 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.005347 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:51.005406 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.006506 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.007557 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.008242 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.008341 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.008834 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.008868 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.008910 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260928 09:04:51.009083 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.004378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44273 (local address 127.0.0.1:52356)
0928 09:04:51.004542 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:51.004553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:51.004602 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:51.005192 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:51.005193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:51.005196 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:51.005232 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.005234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.006598 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.006600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.007764 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.007767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.008848 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.008849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.008910 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.008914 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:51.008968 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:04:51.009007 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260928 09:04:51.009210 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5277d6d9014803b769270628a60f9a (DEFAULT_TABLE table=test-table [id=f5320977613e41169f475db87a1185d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:51.009363 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.004842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.0.98.1:44273)
0928 09:04:51.005003 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.005006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.005015 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:51.005044 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.005046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.005058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.005178 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:51.005297 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.006440 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.007855 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.008662 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.008730 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.008808 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.009228 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.009284 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260928 09:04:51.009632 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5277d6d9014803b769270628a60f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:51.009608 7538 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5277d6d9014803b769270628a60f9a (DEFAULT_TABLE table=test-table [id=f5320977613e41169f475db87a1185d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:51.009931 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5277d6d9014803b769270628a60f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:51.010007 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5277d6d9014803b769270628a60f9a (DEFAULT_TABLE table=test-table [id=f5320977613e41169f475db87a1185d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:51.010365 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5277d6d9014803b769270628a60f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:51.013216 7766 tablet_bootstrap.cc:492] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Bootstrap starting.
I20260928 09:04:51.013741 7767 tablet_bootstrap.cc:492] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Bootstrap starting.
I20260928 09:04:51.014106 7766 tablet_bootstrap.cc:654] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:51.014379 7767 tablet_bootstrap.cc:654] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:51.014428 7766 log.cc:824] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:51.014700 7767 log.cc:824] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:51.014782 7768 tablet_bootstrap.cc:492] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Bootstrap starting.
I20260928 09:04:51.015251 7766 tablet_bootstrap.cc:492] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: No bootstrap required, opened a new log
I20260928 09:04:51.015316 7766 ts_tablet_manager.cc:1397] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 09:04:51.015455 7767 tablet_bootstrap.cc:492] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: No bootstrap required, opened a new log
I20260928 09:04:51.015535 7767 ts_tablet_manager.cc:1397] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:51.017284 7767 raft_consensus.cc:348] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.017444 7767 raft_consensus.cc:374] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:51.017489 7767 raft_consensus.cc:729] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0571bcc2b0054c809593b81342c67d25, State: Initialized, Role: FOLLOWER
I20260928 09:04:51.017660 7767 consensus_queue.cc:260] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.017948 7767 ts_tablet_manager.cc:1428] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:04:51.017979 7740 heartbeater.cc:503] Master 127.0.98.62:38561 was elected leader, sending a full tablet report...
I20260928 09:04:51.017979 7766 raft_consensus.cc:348] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.018139 7766 raft_consensus.cc:374] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:51.018189 7766 raft_consensus.cc:729] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842214b26a4245e7918c6e3051460681, State: Initialized, Role: FOLLOWER
I20260928 09:04:51.018309 7768 tablet_bootstrap.cc:654] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:51.018317 7766 consensus_queue.cc:260] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.018615 7768 log.cc:824] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:51.018970 7766 ts_tablet_manager.cc:1428] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 09:04:51.019122 7467 heartbeater.cc:503] Master 127.0.98.62:38561 was elected leader, sending a full tablet report...
I20260928 09:04:51.019946 7768 tablet_bootstrap.cc:492] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: No bootstrap required, opened a new log
I20260928 09:04:51.020018 7768 ts_tablet_manager.cc:1397] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 09:04:51.022248 7768 raft_consensus.cc:348] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.022373 7768 raft_consensus.cc:374] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:51.022395 7768 raft_consensus.cc:729] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3200bcee5144a6be24f2f1ea138013, State: Initialized, Role: FOLLOWER
I20260928 09:04:51.022500 7768 consensus_queue.cc:260] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.022759 7768 ts_tablet_manager.cc:1428] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:04:51.022828 7604 heartbeater.cc:503] Master 127.0.98.62:38561 was elected leader, sending a full tablet report...
W20260928 09:04:51.086416 7607 tablet.cc:2369] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:51.105086 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.102621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.0.1:58718)
0928 09:04:51.102687 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:51.102704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:51.102745 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:51.102878 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:51.102879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:51.102880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:51.102929 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.102932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.103541 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.103542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.104337 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.104341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.104881 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.104882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.104934 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.104938 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:51.104985 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 09:04:51.105023 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 09:04:51.105085 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.102693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.0.98.60:35241)
0928 09:04:51.102752 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.102754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.102764 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:51.102794 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.102795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.102813 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.102874 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:51.102985 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.103463 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.104413 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.104797 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.104854 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.104908 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.104999 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.105030 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 09:04:51.105443 7301 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 09:04:51.105856 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635e5d7a743d46f2bc6bf623866b6bf2. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:51.105967 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: LEARNER
W20260928 09:04:51.176726 7470 tablet.cc:2369] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:51.181151 7772 raft_consensus.cc:482] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:51.181260 7772 raft_consensus.cc:504] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.181689 7772 leader_election.cc:288] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847), ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133)
I20260928 09:04:51.184597 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.182029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55999 (local address 127.0.98.3:35847)
0928 09:04:51.182075 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.182077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.182087 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:51.182158 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.182159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.182170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.182247 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:51.182413 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.182878 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.183844 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.184236 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.184322 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.184332 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.184490 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.184523 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 09:04:51.184603 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.182010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56259 (local address 127.0.98.2:42133)
0928 09:04:51.182055 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.182058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.182069 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:51.182129 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.182130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.182140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.182214 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:51.182373 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.182838 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.183875 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.184230 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.184323 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.184332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.184486 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.184520 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:04:51.184651 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.181932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35847 (local address 127.0.98.1:55999)
0928 09:04:51.182006 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:51.182020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:51.182086 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:51.182249 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:51.182250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:51.182252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:51.182338 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.182339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.182968 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.182969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.183750 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.183753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.184332 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.184333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.184405 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.184408 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:51.184471 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 09:04:51.184517 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260928 09:04:51.184671 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.181947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42133 (local address 127.0.98.1:56259)
0928 09:04:51.182009 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:51.182020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:51.182069 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:51.182216 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:51.182217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:51.182219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:51.182292 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.182297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.182920 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.182921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.183779 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:51.183781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.184325 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.184326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:51.184395 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.184399 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:51.184480 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 09:04:51.184520 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":44}
I20260928 09:04:51.184999 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5277d6d9014803b769270628a60f9a" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3200bcee5144a6be24f2f1ea138013" is_pre_election: true
I20260928 09:04:51.185000 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5277d6d9014803b769270628a60f9a" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0571bcc2b0054c809593b81342c67d25" is_pre_election: true
I20260928 09:04:51.185158 7694 raft_consensus.cc:2454] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 0.
I20260928 09:04:51.185156 7558 raft_consensus.cc:2454] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 0.
I20260928 09:04:51.185387 7354 leader_election.cc:302] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0571bcc2b0054c809593b81342c67d25, 842214b26a4245e7918c6e3051460681, ea3200bcee5144a6be24f2f1ea138013; no voters:
I20260928 09:04:51.185586 7772 raft_consensus.cc:2781] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:51.185672 7772 raft_consensus.cc:482] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:51.185703 7772 raft_consensus.cc:3037] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:51.186555 7772 raft_consensus.cc:504] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.186715 7772 leader_election.cc:288] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 election: Requested vote from peers 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847), ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133)
I20260928 09:04:51.186868 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5277d6d9014803b769270628a60f9a" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0571bcc2b0054c809593b81342c67d25"
I20260928 09:04:51.186884 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5277d6d9014803b769270628a60f9a" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3200bcee5144a6be24f2f1ea138013"
I20260928 09:04:51.186957 7558 raft_consensus.cc:3037] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:51.186957 7694 raft_consensus.cc:3037] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:51.187855 7694 raft_consensus.cc:2454] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 1.
I20260928 09:04:51.187855 7558 raft_consensus.cc:2454] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 1.
I20260928 09:04:51.188052 7355 leader_election.cc:302] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 842214b26a4245e7918c6e3051460681, ea3200bcee5144a6be24f2f1ea138013; no voters:
I20260928 09:04:51.188198 7772 raft_consensus.cc:2781] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:51.188347 7772 raft_consensus.cc:686] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 1 LEADER]: Becoming Leader. State: Replica: 842214b26a4245e7918c6e3051460681, State: Running, Role: LEADER
I20260928 09:04:51.188437 7772 consensus_queue.cc:237] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } }
I20260928 09:04:51.189504 6496 catalog_manager.cc:5495] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842214b26a4245e7918c6e3051460681 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "842214b26a4245e7918c6e3051460681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:51.190073 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: true } } }
I20260928 09:04:51.190220 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:51.191004 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: attempting to promote NON_VOTER ec7ffeda80154ede9fa30499fea67144 to VOTER
I20260928 09:04:51.191124 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } }
I20260928 09:04:51.191606 7301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 LEARNER]: Refusing update from remote peer 635e5d7a743d46f2bc6bf623866b6bf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 09:04:51.191680 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Refusing update from remote peer 635e5d7a743d46f2bc6bf623866b6bf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 09:04:51.191792 7780 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 09:04:51.191897 7243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 09:04:51.192401 7780 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ec7ffeda80154ede9fa30499fea67144 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.192575 7301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ec7ffeda80154ede9fa30499fea67144 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.192685 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ec7ffeda80154ede9fa30499fea67144 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.193390 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.193528 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.193578 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.193650 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:51.193650 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:04:51.193629 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.193706 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:04:51.193712 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:51.193810 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:51.193576 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2 [sys.catalog]: This master's current role is: LEADER
W20260928 09:04:51.228108 7743 tablet.cc:2369] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:51.235581 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.233045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.0.98.1:44273)
0928 09:04:51.233109 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.233111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.233122 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:51.233195 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.233196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.233216 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.233294 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:51.233435 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.233977 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.234726 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.234923 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.235002 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.235204 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:51.235348 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:51.235443 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.235466 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.235495 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:04:51.237298 7694 raft_consensus.cc:1261] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Refusing update from remote peer 842214b26a4245e7918c6e3051460681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:51.237337 7558 raft_consensus.cc:1261] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Refusing update from remote peer 842214b26a4245e7918c6e3051460681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:51.237713 7772 consensus_queue.cc:1035] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:51.237854 7778 consensus_queue.cc:1035] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:51.240424 7784 mvcc.cc:205] Tried to move back new op lower bound from 7334241448905367552 to 7334241448708546560. Current Snapshot: MvccSnapshot[applied={T|T < 7334241448905367552}]
I20260928 09:04:51.241294 7786 mvcc.cc:205] Tried to move back new op lower bound from 7334241448905367552 to 7334241448708546560. Current Snapshot: MvccSnapshot[applied={T|T < 7334241448905367552}]
I20260928 09:04:51.242251 7785 mvcc.cc:205] Tried to move back new op lower bound from 7334241448905367552 to 7334241448708546560. Current Snapshot: MvccSnapshot[applied={T|T < 7334241448905367552 or (T in {7334241448905367552})}]
I20260928 09:04:51.243393 6497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:51.243484 6497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:51.244797 6497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 635e5d7a743d46f2bc6bf623866b6bf2: Sending DeleteTablet for 3 replicas of tablet 0c5277d6d9014803b769270628a60f9a
I20260928 09:04:51.245141 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5277d6d9014803b769270628a60f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41138
I20260928 09:04:51.245253 7538 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5277d6d9014803b769270628a60f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51694
I20260928 09:04:51.245525 7796 tablet_replica.cc:330] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: stopping tablet replica
I20260928 09:04:51.245661 7796 raft_consensus.cc:2229] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:51.245787 7796 raft_consensus.cc:2258] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:51.245986 7795 tablet_replica.cc:330] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: stopping tablet replica
I20260928 09:04:51.245990 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5277d6d9014803b769270628a60f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52356
I20260928 09:04:51.246099 7795 raft_consensus.cc:2229] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:51.246210 7795 raft_consensus.cc:2258] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:51.246378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6465
I20260928 09:04:51.247449 7796 ts_tablet_manager.cc:1905] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:51.247505 7795 ts_tablet_manager.cc:1905] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:51.249238 7795 ts_tablet_manager.cc:1918] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:51.249317 7795 log.cc:1196] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/0c5277d6d9014803b769270628a60f9a
I20260928 09:04:51.249452 7796 ts_tablet_manager.cc:1918] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:51.249571 7796 log.cc:1196] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/0c5277d6d9014803b769270628a60f9a
I20260928 09:04:51.249666 7795 ts_tablet_manager.cc:1939] T 0c5277d6d9014803b769270628a60f9a P 0571bcc2b0054c809593b81342c67d25: Deleting consensus metadata
I20260928 09:04:51.249840 7796 ts_tablet_manager.cc:1939] T 0c5277d6d9014803b769270628a60f9a P ea3200bcee5144a6be24f2f1ea138013: Deleting consensus metadata
W20260928 09:04:51.252942 7492 connection.cc:535] server connection from 127.0.0.1:51694 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 09:04:51.252972 7630 connection.cc:535] server connection from 127.0.0.1:41138 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:04:51.253672 7798 tablet_replica.cc:330] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: stopping tablet replica
I20260928 09:04:51.253796 7798 raft_consensus.cc:2229] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:51.253921 7798 raft_consensus.cc:2258] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:51.254400 7798 ts_tablet_manager.cc:1905] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:51.256371 7798 ts_tablet_manager.cc:1918] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:51.256455 7798 log.cc:1196] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/0c5277d6d9014803b769270628a60f9a
I20260928 09:04:51.256762 7798 ts_tablet_manager.cc:1939] T 0c5277d6d9014803b769270628a60f9a P 842214b26a4245e7918c6e3051460681: Deleting consensus metadata
W20260928 09:04:51.257083 7357 connection.cc:441] server connection from 127.0.0.1:52356 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52356 (request call id 1) could send its response
I20260928 09:04:51.262657 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.245475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.0.98.60:35241)
0928 09:04:51.245685 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.245687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.245695 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:51.245887 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.245888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.245899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.245931 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:51.255285 (+ 9354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.256072 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.257139 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.257430 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.257472 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.259163 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:51.259164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:51.259164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:51.259171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:51.261344 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:51.261761 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:51.261762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:51.261762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:51.261823 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:51.262062 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:51.262063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:51.262064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:51.262186 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:51.262267 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.262459 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.262539 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260928 09:04:51.263629 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.254683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.0.98.60:35241)
0928 09:04:51.254725 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.254727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.255077 (+ 350us) server_negotiation.cc:407] Connection header received
0928 09:04:51.255179 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.255180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.255213 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.255245 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:51.256765 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.257390 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.258945 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.259215 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.259257 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.260635 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:51.260636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:51.260636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:51.260644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:51.262254 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:51.262712 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:51.262713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:51.262713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:51.262771 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:51.262983 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:51.262984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:51.262984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:51.263123 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:51.263209 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.263464 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.263535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:51.269874 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:51.255130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.98.61:37361)
0928 09:04:51.255175 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:51.255177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:51.263661 (+ 8484us) server_negotiation.cc:407] Connection header received
0928 09:04:51.263868 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:51.263869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:51.263880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:51.263931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:51.264400 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.264867 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.265535 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:51.265782 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:51.265866 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:51.267285 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:51.267286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:51.267287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:51.267295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:51.268663 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:51.269047 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:51.269048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:51.269049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:51.269110 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:51.269298 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:51.269299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:51.269299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:51.269413 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:51.269510 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:51.269687 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 09:04:51.269762 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:52.242316 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.241833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:55365)
0928 09:04:52.242107 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.242254 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
W20260928 09:04:52.242533 7467 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.242998 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.242313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:37677)
0928 09:04:52.242827 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.242893 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":50,"threads_started":1}
I20260928 09:04:52.243158 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.242364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:52789)
0928 09:04:52.243036 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.243107 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":46,"threads_started":1}
W20260928 09:04:52.243253 7604 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
W20260928 09:04:52.243346 7740 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.745651 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 635e5d7a743d46f2bc6bf623866b6bf2)
I20260928 09:04:52.745738 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } }
I20260928 09:04:52.745976 7808 leader_election.cc:288] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561), 98bce89f6168420d89392bff35393b3f (127.0.98.61:37361)
I20260928 09:04:52.746294 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 635e5d7a743d46f2bc6bf623866b6bf2)
I20260928 09:04:52.746506 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.746261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39892)
0928 09:04:52.746356 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.746422 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20260928 09:04:52.746479 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } }
W20260928 09:04:52.746732 7272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.747001 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561), ec7ffeda80154ede9fa30499fea67144 (127.0.98.60:35241)
I20260928 09:04:52.747193 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.746849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39904)
0928 09:04:52.747106 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.747156 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":47,"threads_started":1}
W20260928 09:04:52.747288 7079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.747354 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.747263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39916)
0928 09:04:52.747297 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.747327 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 09:04:52.747500 7272 leader_election.cc:334] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.747732 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.747610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39930)
0928 09:04:52.747663 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.747704 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 09:04:52.747900 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.749747 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.746196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.0.98.61:37361)
0928 09:04:52.746297 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:52.746300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:52.746436 (+ 136us) server_negotiation.cc:407] Connection header received
0928 09:04:52.746547 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:52.746549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:52.746560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:52.746609 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:52.746826 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.747595 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.749013 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.749397 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.749487 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.749499 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:52.749643 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 09:04:52.749684 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":36,"threads_started":1}
I20260928 09:04:52.749816 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.746281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37361 (local address 127.0.0.1:41616)
0928 09:04:52.746364 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.746374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:52.746464 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:52.746670 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:52.746670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:52.746672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:52.746715 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.746716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.747701 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.747702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.748920 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.748923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.749489 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.749490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.749576 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.749580 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:52.749637 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:04:52.749736 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":50,"thread_start_us":38,"threads_started":1}
I20260928 09:04:52.750028 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec7ffeda80154ede9fa30499fea67144" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98bce89f6168420d89392bff35393b3f" is_pre_election: true
I20260928 09:04:52.750123 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec7ffeda80154ede9fa30499fea67144 in term 1.
I20260928 09:04:52.750273 7275 leader_election.cc:302] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98bce89f6168420d89392bff35393b3f, ec7ffeda80154ede9fa30499fea67144; no voters: 635e5d7a743d46f2bc6bf623866b6bf2
I20260928 09:04:52.750370 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 09:04:52.750447 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 635e5d7a743d46f2bc6bf623866b6bf2)
I20260928 09:04:52.750478 7808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:04:52.750488 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.746867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35241 (local address 127.0.0.1:60264)
0928 09:04:52.747495 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.747505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:52.747656 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:52.747829 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:52.747830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:52.747832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:52.747869 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.747870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.748730 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.748731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.749460 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.749462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.750233 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.750233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.750303 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.750305 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:04:52.750379 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:04:52.750418 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":100,"threads_started":1}
I20260928 09:04:52.750706 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.747070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.0.98.60:35241)
0928 09:04:52.747648 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:52.747650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:52.747659 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:52.747733 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:52.747734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:52.747745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:52.747811 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:52.747959 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.748623 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.749584 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.750122 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.750193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.750205 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:52.750609 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 09:04:52.750649 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":78,"threads_started":1}
I20260928 09:04:52.750880 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98bce89f6168420d89392bff35393b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec7ffeda80154ede9fa30499fea67144" is_pre_election: true
I20260928 09:04:52.751340 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } }
I20260928 09:04:52.751498 7301 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98bce89f6168420d89392bff35393b3f in current term 2: Already voted for candidate ec7ffeda80154ede9fa30499fea67144 in this term.
I20260928 09:04:52.751502 7808 leader_election.cc:288] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 2 election: Requested vote from peers 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561), 98bce89f6168420d89392bff35393b3f (127.0.98.61:37361)
I20260928 09:04:52.751627 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec7ffeda80154ede9fa30499fea67144" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98bce89f6168420d89392bff35393b3f"
I20260928 09:04:52.751694 7105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:04:52.751716 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98bce89f6168420d89392bff35393b3f; no voters: 635e5d7a743d46f2bc6bf623866b6bf2, ec7ffeda80154ede9fa30499fea67144
I20260928 09:04:52.752351 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec7ffeda80154ede9fa30499fea67144 in term 2.
I20260928 09:04:52.752457 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:04:52.752533 7275 leader_election.cc:302] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98bce89f6168420d89392bff35393b3f, ec7ffeda80154ede9fa30499fea67144; no voters:
I20260928 09:04:52.752537 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.751566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39954)
0928 09:04:52.752477 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.752509 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":894}
I20260928 09:04:52.752635 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:04:52.752776 7808 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [term 2 LEADER]: Becoming Leader. State: Replica: ec7ffeda80154ede9fa30499fea67144, State: Running, Role: LEADER
I20260928 09:04:52.752868 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.752796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39962)
0928 09:04:52.752831 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.752850 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 09:04:52.752861 7808 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } }
W20260928 09:04:52.752985 7272 leader_election.cc:334] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
I20260928 09:04:52.753237 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7ffeda80154ede9fa30499fea67144. Latest consensus state: current_term: 2 leader_uuid: "ec7ffeda80154ede9fa30499fea67144" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:52.753340 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:52.753594 7820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:52.754026 7820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:52.754148 7820 catalog_manager.cc:1259] Loaded cluster ID: 33dd3e9fcb7f4bb98e93f1e8431b678f
I20260928 09:04:52.754179 7820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:52.754350 7820 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:52.754469 7820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Loaded TSK: 0
I20260928 09:04:52.754637 7820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:04:52.804222 7290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:52.805892 7290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:52.806857 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.806747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:39986)
0928 09:04:52.806792 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.806831 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 09:04:52.806885 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [term 2 FOLLOWER]: Refusing update from remote peer ec7ffeda80154ede9fa30499fea67144: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 09:04:52.807159 7808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 09:04:52.807235 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.807112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.0.1:40002)
0928 09:04:52.807194 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.807218 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 09:04:52.807420 7272 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 -> Peer 635e5d7a743d46f2bc6bf623866b6bf2 (127.0.98.62:38561): Couldn't send request to peer 635e5d7a743d46f2bc6bf623866b6bf2. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 09:04:52.807794 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7ffeda80154ede9fa30499fea67144. Latest consensus state: current_term: 2 leader_uuid: "ec7ffeda80154ede9fa30499fea67144" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:52.807890 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:04:52.808341 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ec7ffeda80154ede9fa30499fea67144" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:52.808458 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:52.808519 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ec7ffeda80154ede9fa30499fea67144" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "635e5d7a743d46f2bc6bf623866b6bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38561 } } peers { permanent_uuid: "98bce89f6168420d89392bff35393b3f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37361 } attrs { promote: false } } peers { permanent_uuid: "ec7ffeda80154ede9fa30499fea67144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35241 } attrs { promote: false } } }
I20260928 09:04:52.808607 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bce89f6168420d89392bff35393b3f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:04:52.813345 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.810409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.0.98.2:42133)
0928 09:04:52.810511 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:52.810513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:52.810613 (+ 100us) server_negotiation.cc:407] Connection header received
0928 09:04:52.810639 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:52.810640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:52.810651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:52.810708 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:52.810840 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.811541 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.812449 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.812976 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.813077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.813091 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:52.813237 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 09:04:52.813273 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260928 09:04:52.813345 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.810220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42133 (local address 127.0.0.1:37418)
0928 09:04:52.810509 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.810520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:52.810565 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:52.810724 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:52.810724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:52.810727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:52.810761 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.810762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.811609 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.811610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.812348 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.812350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.813079 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.813080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.813141 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.813145 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:52.813229 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:04:52.813273 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260928 09:04:52.813598 7538 tablet_service.cc:1467] Processing CreateTablet for tablet 3cec87464e154587b4f597b020db7932 (DEFAULT_TABLE table=test-table [id=53194c109b7e48438e5233eb66dad7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:52.813722 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cec87464e154587b4f597b020db7932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:52.813764 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.810124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44273 (local address 127.0.0.1:40236)
0928 09:04:52.810156 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.810166 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:52.810206 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:52.810664 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:52.810665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:52.810666 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:52.810706 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.810708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.811391 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.811392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.812539 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.812542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.813562 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.813563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.813626 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.813631 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:52.813660 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 09:04:52.813703 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 09:04:52.813910 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.810207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35847 (local address 127.0.0.1:36046)
0928 09:04:52.810344 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.810354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:52.810398 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:52.810559 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:52.810560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:52.810563 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:52.810622 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.810624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.811556 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.811556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.812620 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:52.812622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.813719 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.813720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:52.813778 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.813781 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:04:52.813813 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 09:04:52.813854 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 09:04:52.814050 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.810181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.98.1:44273)
0928 09:04:52.810287 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:52.810289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:52.810299 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:52.810327 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:52.810328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:52.810338 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:52.810774 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:52.810824 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.811296 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.812725 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.813445 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.813878 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.813894 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:52.813918 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 09:04:52.813967 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260928 09:04:52.814260 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 3cec87464e154587b4f597b020db7932 (DEFAULT_TABLE table=test-table [id=53194c109b7e48438e5233eb66dad7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:52.814399 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cec87464e154587b4f597b020db7932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:52.814520 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.810301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.0.98.3:35847)
0928 09:04:52.810410 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:52.810413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:52.810420 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:52.810447 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:52.810448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:52.810459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:52.810551 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:52.810707 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.811468 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.812707 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:52.813311 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:52.814348 (+ 1037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:52.814372 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:52.814392 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 09:04:52.814448 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260928 09:04:52.815274 7831 tablet_bootstrap.cc:492] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Bootstrap starting.
I20260928 09:04:52.815676 7831 tablet_bootstrap.cc:654] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:52.815694 7832 tablet_bootstrap.cc:492] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681: Bootstrap starting.
I20260928 09:04:52.815840 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 3cec87464e154587b4f597b020db7932 (DEFAULT_TABLE table=test-table [id=53194c109b7e48438e5233eb66dad7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:52.815963 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cec87464e154587b4f597b020db7932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:52.816066 7832 tablet_bootstrap.cc:654] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:52.816295 7831 tablet_bootstrap.cc:492] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: No bootstrap required, opened a new log
I20260928 09:04:52.816356 7831 ts_tablet_manager.cc:1397] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:52.816490 7831 raft_consensus.cc:348] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:52.816578 7831 raft_consensus.cc:374] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:52.816612 7831 raft_consensus.cc:729] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3200bcee5144a6be24f2f1ea138013, State: Initialized, Role: FOLLOWER
I20260928 09:04:52.816603 7832 tablet_bootstrap.cc:492] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681: No bootstrap required, opened a new log
I20260928 09:04:52.816646 7832 ts_tablet_manager.cc:1397] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:52.816641 7831 consensus_queue.cc:260] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:52.816782 7831 ts_tablet_manager.cc:1428] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:52.816759 7832 raft_consensus.cc:348] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:52.816824 7832 raft_consensus.cc:374] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:52.816860 7832 raft_consensus.cc:729] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842214b26a4245e7918c6e3051460681, State: Initialized, Role: FOLLOWER
I20260928 09:04:52.816933 7832 consensus_queue.cc:260] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:52.817062 7832 ts_tablet_manager.cc:1428] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:52.817631 7468 heartbeater.cc:503] Master 127.0.98.60:35241 was elected leader, sending a full tablet report...
I20260928 09:04:52.817921 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.816999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:42119)
0928 09:04:52.817675 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.817752 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":91,"threads_started":1}
I20260928 09:04:52.818289 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.817781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:35467)
0928 09:04:52.818219 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.818253 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":94,"threads_started":1}
I20260928 09:04:52.818406 7838 tablet_bootstrap.cc:492] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Bootstrap starting.
I20260928 09:04:52.818661 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.818559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:38201)
0928 09:04:52.818610 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.818639 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 09:04:52.818814 7838 tablet_bootstrap.cc:654] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:52.819093 7605 heartbeater.cc:503] Master 127.0.98.60:35241 was elected leader, sending a full tablet report...
I20260928 09:04:52.819264 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.819176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:38857)
0928 09:04:52.819214 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.819237 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 09:04:52.819594 7838 tablet_bootstrap.cc:492] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: No bootstrap required, opened a new log
I20260928 09:04:52.819638 7838 ts_tablet_manager.cc:1397] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:52.819725 7838 raft_consensus.cc:348] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:52.819772 7838 raft_consensus.cc:374] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:52.819785 7838 raft_consensus.cc:729] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0571bcc2b0054c809593b81342c67d25, State: Initialized, Role: FOLLOWER
I20260928 09:04:52.819842 7838 consensus_queue.cc:260] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:52.819994 7838 ts_tablet_manager.cc:1428] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:52.820469 7741 heartbeater.cc:503] Master 127.0.98.60:35241 was elected leader, sending a full tablet report...
I20260928 09:04:52.820470 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.820187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:51339)
0928 09:04:52.820395 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.820434 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
I20260928 09:04:52.820915 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:52.820761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:53457)
0928 09:04:52.820876 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:52.820897 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 09:04:53.080011 7836 raft_consensus.cc:482] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:53.080116 7836 raft_consensus.cc:504] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:53.080318 7836 leader_election.cc:288] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133), 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847)
I20260928 09:04:53.080611 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cec87464e154587b4f597b020db7932" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3200bcee5144a6be24f2f1ea138013" is_pre_election: true
I20260928 09:04:53.080612 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cec87464e154587b4f597b020db7932" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0571bcc2b0054c809593b81342c67d25" is_pre_election: true
I20260928 09:04:53.080715 7694 raft_consensus.cc:2454] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 0.
I20260928 09:04:53.080715 7558 raft_consensus.cc:2454] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 0.
I20260928 09:04:53.080895 7354 leader_election.cc:302] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0571bcc2b0054c809593b81342c67d25, 842214b26a4245e7918c6e3051460681; no voters:
I20260928 09:04:53.081063 7836 raft_consensus.cc:2781] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:53.081148 7836 raft_consensus.cc:482] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:53.081178 7836 raft_consensus.cc:3037] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:53.081919 7836 raft_consensus.cc:504] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:53.082082 7836 leader_election.cc:288] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 election: Requested vote from peers ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133), 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847)
I20260928 09:04:53.082257 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cec87464e154587b4f597b020db7932" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0571bcc2b0054c809593b81342c67d25"
I20260928 09:04:53.082257 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cec87464e154587b4f597b020db7932" candidate_uuid: "842214b26a4245e7918c6e3051460681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3200bcee5144a6be24f2f1ea138013"
I20260928 09:04:53.082340 7694 raft_consensus.cc:3037] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:53.082340 7558 raft_consensus.cc:3037] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:53.083115 7694 raft_consensus.cc:2454] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 1.
I20260928 09:04:53.083115 7558 raft_consensus.cc:2454] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842214b26a4245e7918c6e3051460681 in term 1.
I20260928 09:04:53.083302 7355 leader_election.cc:302] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 842214b26a4245e7918c6e3051460681, ea3200bcee5144a6be24f2f1ea138013; no voters:
I20260928 09:04:53.083405 7836 raft_consensus.cc:2781] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:53.083554 7836 raft_consensus.cc:686] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [term 1 LEADER]: Becoming Leader. State: Replica: 842214b26a4245e7918c6e3051460681, State: Running, Role: LEADER
I20260928 09:04:53.083598 7836 consensus_queue.cc:237] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } }
I20260928 09:04:53.084126 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.083785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:60071)
0928 09:04:53.084051 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.084091 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260928 09:04:53.084556 7290 catalog_manager.cc:5495] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842214b26a4245e7918c6e3051460681 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "842214b26a4245e7918c6e3051460681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842214b26a4245e7918c6e3051460681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:53.095116 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.092993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.0.98.1:44273)
0928 09:04:53.093036 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:53.093038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:53.093079 (+ 41us) server_negotiation.cc:407] Connection header received
0928 09:04:53.093100 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:53.093101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:53.093112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:53.093153 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:53.093271 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:53.093815 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:53.094555 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:53.094753 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:53.094825 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:53.094919 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:53.094944 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:53.094979 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:53.095043 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 09:04:53.095060 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:04:53.096117 7694 raft_consensus.cc:1261] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Refusing update from remote peer 842214b26a4245e7918c6e3051460681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:53.096158 7558 raft_consensus.cc:1261] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Refusing update from remote peer 842214b26a4245e7918c6e3051460681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:53.096362 7836 consensus_queue.cc:1035] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0571bcc2b0054c809593b81342c67d25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:53.096482 7843 consensus_queue.cc:1035] T 3cec87464e154587b4f597b020db7932 P 842214b26a4245e7918c6e3051460681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3200bcee5144a6be24f2f1ea138013" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:53.097338 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.097207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:51295)
0928 09:04:53.097260 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.097292 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:04:53.097375 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.097249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:40619)
0928 09:04:53.097306 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.097343 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:04:53.098496 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.098402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:36075)
0928 09:04:53.098440 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.098465 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 09:04:53.098510 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.098421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.3:43973)
0928 09:04:53.098452 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.098480 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 09:04:53.099952 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.099815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.1:59859)
0928 09:04:53.099896 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.099921 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 09:04:53.100144 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:53.100045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38561 (local address 127.0.98.2:42363)
0928 09:04:53.100080 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.100112 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 09:04:53.100585 7290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:53.100679 7290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:53.101943 7290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec7ffeda80154ede9fa30499fea67144: Sending DeleteTablet for 3 replicas of tablet 3cec87464e154587b4f597b020db7932
I20260928 09:04:53.102320 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cec87464e154587b4f597b020db7932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:53 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36046
I20260928 09:04:53.102378 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cec87464e154587b4f597b020db7932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:53 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40236
I20260928 09:04:53.102564 7538 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cec87464e154587b4f597b020db7932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:53 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37418
I20260928 09:04:53.102623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 7242
I20260928 09:04:53.103462 7857 tablet_replica.cc:330] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: stopping tablet replica
I20260928 09:04:53.103542 7857 raft_consensus.cc:2229] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:53.103602 7857 raft_consensus.cc:2258] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:53.103729 7855 tablet_replica.cc:330] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: stopping tablet replica
I20260928 09:04:53.103785 7855 raft_consensus.cc:2229] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:53.103909 7855 raft_consensus.cc:2258] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:53.104035 7857 ts_tablet_manager.cc:1905] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:53.104239 7855 ts_tablet_manager.cc:1905] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:53.105916 7855 ts_tablet_manager.cc:1918] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:53.105916 7857 ts_tablet_manager.cc:1918] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:53.105978 7857 log.cc:1196] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/3cec87464e154587b4f597b020db7932
I20260928 09:04:53.105978 7855 log.cc:1196] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/3cec87464e154587b4f597b020db7932
I20260928 09:04:53.106225 7855 ts_tablet_manager.cc:1939] T 3cec87464e154587b4f597b020db7932 P 0571bcc2b0054c809593b81342c67d25: Deleting consensus metadata
I20260928 09:04:53.106225 7857 ts_tablet_manager.cc:1939] T 3cec87464e154587b4f597b020db7932 P ea3200bcee5144a6be24f2f1ea138013: Deleting consensus metadata
I20260928 09:04:53.106521 7273 catalog_manager.cc:4841] TS ea3200bcee5144a6be24f2f1ea138013 (127.0.98.2:42133): tablet 3cec87464e154587b4f597b020db7932 (table test-table [id=53194c109b7e48438e5233eb66dad7b8]) successfully deleted
I20260928 09:04:53.106572 7272 catalog_manager.cc:4841] TS 0571bcc2b0054c809593b81342c67d25 (127.0.98.3:35847): tablet 3cec87464e154587b4f597b020db7932 (table test-table [id=53194c109b7e48438e5233eb66dad7b8]) successfully deleted
W20260928 09:04:53.109491 7811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:53.109349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44273 (local address 127.0.0.1:40264)
0928 09:04:53.109407 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:53.109437 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44273: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 09:04:53.109661 7273 catalog_manager.cc:4584] TS 842214b26a4245e7918c6e3051460681 (127.0.98.1:44273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cec87464e154587b4f597b020db7932: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44273: connect: Connection reset by peer (error 104)
I20260928 09:04:53.109685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 7475
I20260928 09:04:53.115530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 7611
I20260928 09:04:53.120235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 6961
I20260928 09:04:53.124976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 7149
2026-09-28T09:04:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4991 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:53 dist-test-slave-smns krb5kdc[7860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:53 dist-test-slave-smns krb5kdc[7860](info): set up 2 sockets
Sep 28 09:04:53 dist-test-slave-smns krb5kdc[7860](info): commencing operation
krb5kdc: starting...
W20260928 09:04:54.161993 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 09:04:54.280916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:54.362104 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:54.362306 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:54.362329 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:54.363444 7873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:54.363485 7873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 09:04:54.363499 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:54.363512 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:54.363524 7873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:54.363538 7873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:54.364924 7873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34941
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:54.365111 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:54.365268 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:54.370240 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:54.370265 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:54.370318 7873 server_base.cc:1034] running on GCE node
W20260928 09:04:54.370286 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:54.370654 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:54.370929 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:54.372073 7873 hybrid_clock.cc:648] HybridClock initialized: now 1790586294372047 us; error 38 us; skew 500 ppm
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:54.374670 7873 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 09:04:54.375113 7873 webserver.cc:460] Webserver started at http://127.0.98.62:43175/ using document root <none> and password file <none>
I20260928 09:04:54.375313 7873 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:54.375355 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:54.375448 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:54.376497 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "31eac824aa114125bc3151a022f950a0"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "138ec39bb213a996380aabdb433262ec"
server_key_iv: "2f1c87e407571481680511bb495c851d"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.376791 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "31eac824aa114125bc3151a022f950a0"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "138ec39bb213a996380aabdb433262ec"
server_key_iv: "2f1c87e407571481680511bb495c851d"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.378109 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:54.378782 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.378943 7873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:54.379022 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "31eac824aa114125bc3151a022f950a0"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "138ec39bb213a996380aabdb433262ec"
server_key_iv: "2f1c87e407571481680511bb495c851d"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.379071 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:54.394744 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:54.396678 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:54.396785 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:54.401237 7873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34941
I20260928 09:04:54.401353 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34941 every 8 connection(s)
I20260928 09:04:54.401712 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:54.402428 7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:54.404722 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Bootstrap starting.
I20260928 09:04:54.405320 7951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:54.405398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 7873
I20260928 09:04:54.405519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:54.405714 392 external_mini_cluster.cc:1428] Setting key 39a4e9b1983983bc122081f1691848c6
I20260928 09:04:54.405702 7951 log.cc:824] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:54.406539 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: No bootstrap required, opened a new log
I20260928 09:04:54.408499 7951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eac824aa114125bc3151a022f950a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34941 } }
I20260928 09:04:54.408634 7951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:54.408658 7951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31eac824aa114125bc3151a022f950a0, State: Initialized, Role: FOLLOWER
I20260928 09:04:54.408790 7951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eac824aa114125bc3151a022f950a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34941 } }
I20260928 09:04:54.408869 7951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:54.408910 7951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:54.408960 7951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:54.409586 7951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eac824aa114125bc3151a022f950a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34941 } }
I20260928 09:04:54.409695 7951 leader_election.cc:302] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31eac824aa114125bc3151a022f950a0; no voters:
I20260928 09:04:54.409857 7951 leader_election.cc:288] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:54.409904 7956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:54.410104 7956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [term 1 LEADER]: Becoming Leader. State: Replica: 31eac824aa114125bc3151a022f950a0, State: Running, Role: LEADER
I20260928 09:04:54.410111 7951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:54.410198 7956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eac824aa114125bc3151a022f950a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34941 } }
I20260928 09:04:54.410521 7956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31eac824aa114125bc3151a022f950a0. Latest consensus state: current_term: 1 leader_uuid: "31eac824aa114125bc3151a022f950a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eac824aa114125bc3151a022f950a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34941 } } }
I20260928 09:04:54.410598 7956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:54.410533 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31eac824aa114125bc3151a022f950a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eac824aa114125bc3151a022f950a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34941 } } }
I20260928 09:04:54.410638 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0 [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:54.410768 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:54.411237 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:54.413847 7962 catalog_manager.cc:1347] Generated new cluster ID: f3a69050ad3a469695410e34c71a05b0
I20260928 09:04:54.413888 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:54.418207 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.406249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.0.98.62:34941)
0928 09:04:54.406420 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.406429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.406455 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:04:54.406932 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.406939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.406988 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.407101 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.407902 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.409350 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.410482 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.411105 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.411209 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.414371 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.414375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.414375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.414386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.416297 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.416715 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.416718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.416719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.416813 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.416982 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.416983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.416984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.417247 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.417378 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.417867 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.418006 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 09:04:54.441234 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:54.442018 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:54.445919 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Generated new TSK 0
I20260928 09:04:54.446142 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 09:04:54.478608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34941
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:54.559849 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:54.560093 7977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:54.560132 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:54.561806 7977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:54.561865 7977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:54.561902 7977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:54.561946 7977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:54.563498 7977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:54.563759 7977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:54.564007 7977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:54.569114 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:54.569149 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:54.569195 7977 server_base.cc:1034] running on GCE node
W20260928 09:04:54.569149 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:54.569583 7977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:54.569803 7977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:54.570964 7977 hybrid_clock.cc:648] HybridClock initialized: now 1790586294570954 us; error 31 us; skew 500 ppm
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:54.573340 7977 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 09:04:54.573820 7977 webserver.cc:460] Webserver started at http://127.0.98.1:39605/ using document root <none> and password file <none>
I20260928 09:04:54.574080 7977 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:54.574127 7977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:54.574252 7977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:54.575341 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "d256718298d347659b4a8bcfd60974e7"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "4c4152e8cad28f0a8969b5c6b2712bcc"
server_key_iv: "edcf18e6abf98b8241470af581421c12"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.575735 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d256718298d347659b4a8bcfd60974e7"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "4c4152e8cad28f0a8969b5c6b2712bcc"
server_key_iv: "edcf18e6abf98b8241470af581421c12"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.577062 7977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:04:54.577912 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.578094 7977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:54.578199 7977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "d256718298d347659b4a8bcfd60974e7"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "4c4152e8cad28f0a8969b5c6b2712bcc"
server_key_iv: "edcf18e6abf98b8241470af581421c12"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.578271 7977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:54.593245 7977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:54.595146 7977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:54.595315 7977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:54.595866 7977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:54.595933 7977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.595991 7977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:54.596010 7977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:54.603400 7977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34921
I20260928 09:04:54.603758 7977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:54.603749 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34921 every 8 connection(s)
I20260928 09:04:54.605743 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.596267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57903 (local address 127.0.98.62:34941)
0928 09:04:54.596325 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.596328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.596608 (+ 280us) server_negotiation.cc:407] Connection header received
0928 09:04:54.597138 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.597139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.597153 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.597210 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.597963 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.598528 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.599640 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.599950 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.600049 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.602602 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.602603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.602603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.602612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.604293 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.604727 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.604728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.604728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.604795 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.605058 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.605059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.605060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.605185 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.605256 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.605564 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.605651 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:04:54.605930 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.596263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.98.1:57903)
0928 09:04:54.596422 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.596456 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.597024 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.597290 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.597293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.597668 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:54.597858 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.597864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.598626 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.598627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.599480 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.599482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.600055 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.600057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.600188 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.600771 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:54.600783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:54.602499 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:54.604397 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.604403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.604405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.604635 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.604863 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.604864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.604864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.604978 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.605258 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:54.605264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:54.605405 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.605516 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.605695 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260928 09:04:54.606473 8108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34941
I20260928 09:04:54.606565 8108 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:54.606774 8108 heartbeater.cc:511] Master 127.0.98.62:34941 requested a full tablet report, sending...
I20260928 09:04:54.607304 7904 ts_manager.cc:194] Registered new tserver with Master: d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921)
I20260928 09:04:54.608146 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57903
I20260928 09:04:54.612771 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.607682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52661 (local address 127.0.98.62:34941)
0928 09:04:54.607721 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.607722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.607851 (+ 129us) server_negotiation.cc:407] Connection header received
0928 09:04:54.607877 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.607878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.607888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.607951 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.608214 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.608784 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.609481 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.609718 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.609769 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.610742 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.610743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.610744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.610749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.611924 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.612218 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.612219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.612219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.612265 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.612420 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.612425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.612425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.612513 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.612560 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.612664 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.612713 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:04:54.612788 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.607609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.98.1:52661)
0928 09:04:54.607777 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.607788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.607839 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.607954 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.607954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.608101 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:54.608149 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.608151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.608880 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.608881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.609410 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.609412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.609781 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.609782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.609892 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.610375 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:54.610380 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:54.610667 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:54.611997 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.611997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.611998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.612153 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.612328 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.612328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.612329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.612373 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.612567 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:54.612568 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:54.612601 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.612650 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.612722 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260928 09:04:54.613776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 7977
I20260928 09:04:54.613897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:54.614132 392 external_mini_cluster.cc:1428] Setting key 666b78c2e0f8a520a3439fec985b01e6
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 09:04:54.641026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34941
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:54.722863 8117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:54.723129 8117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:54.723155 8117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:54.724370 8117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:54.724416 8117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:54.724436 8117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:54.724490 8117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:54.726099 8117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:54.726333 8117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:54.726584 8117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:54.731738 8122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:54.731761 8123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:54.731769 8125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:54.732118 8117 server_base.cc:1034] running on GCE node
I20260928 09:04:54.732326 8117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:54.732595 8117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:54.733759 8117 hybrid_clock.cc:648] HybridClock initialized: now 1790586294733744 us; error 24 us; skew 500 ppm
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:54.736511 8117 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 09:04:54.737032 8117 webserver.cc:460] Webserver started at http://127.0.98.2:34491/ using document root <none> and password file <none>
I20260928 09:04:54.737269 8117 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:54.737316 8117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:54.737474 8117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:54.738557 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae9168cd27a9458cafdde6f28beedd6e"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "f6f8ddc7e1eb35630679f1430b60cb28"
server_key_iv: "c944835c5651ed20efc9229bd4bf01c2"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.738917 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae9168cd27a9458cafdde6f28beedd6e"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "f6f8ddc7e1eb35630679f1430b60cb28"
server_key_iv: "c944835c5651ed20efc9229bd4bf01c2"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.740234 8117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:04:54.740967 8132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.741140 8117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:54.741238 8117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "ae9168cd27a9458cafdde6f28beedd6e"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "f6f8ddc7e1eb35630679f1430b60cb28"
server_key_iv: "c944835c5651ed20efc9229bd4bf01c2"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.741321 8117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:54.761480 8117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:54.763957 8117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:54.764119 8117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:54.764710 8117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:54.764788 8117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.764855 8117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:54.764875 8117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:54.772264 8117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44921
I20260928 09:04:54.772341 8247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44921 every 8 connection(s)
I20260928 09:04:54.772681 8117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:54.774305 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.765144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51535 (local address 127.0.98.62:34941)
0928 09:04:54.765198 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.765200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.765429 (+ 229us) server_negotiation.cc:407] Connection header received
0928 09:04:54.766116 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.766118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.766133 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.766208 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.766917 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.767440 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.768502 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.768762 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.768846 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.771455 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.771457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.771457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.771464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.772943 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.773349 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.773350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.773350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.773406 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.773723 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.773724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.773724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.773834 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.773891 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.774130 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.774234 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 09:04:54.774375 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.765131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.98.2:51535)
0928 09:04:54.765274 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.765334 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.765995 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.766301 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.766304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.766585 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:54.766828 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.766833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.767537 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.767539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.768408 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.768412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.768856 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.768858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.769005 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.769524 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:54.769534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:54.771333 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:54.773035 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.773040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.773042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.773260 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.773504 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.773505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.773505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.773661 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.773895 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:54.773899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:54.774029 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.774104 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.774217 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 09:04:54.774828 8248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34941
I20260928 09:04:54.774926 8248 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:54.775113 8248 heartbeater.cc:511] Master 127.0.98.62:34941 requested a full tablet report, sending...
I20260928 09:04:54.775540 7905 ts_manager.cc:194] Registered new tserver with Master: ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921)
I20260928 09:04:54.775882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 8117
I20260928 09:04:54.775985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:54.776070 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51535
I20260928 09:04:54.776199 392 external_mini_cluster.cc:1428] Setting key dcd2f7edcbc11f492c53db69214ae102
I20260928 09:04:54.782939 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.776071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44065 (local address 127.0.98.62:34941)
0928 09:04:54.776192 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.776194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.776250 (+ 56us) server_negotiation.cc:407] Connection header received
0928 09:04:54.776332 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.776333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.776344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.776405 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.777337 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.777947 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.778757 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.778960 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.779044 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.779997 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.779998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.779998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.780004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.781451 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.782043 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.782044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.782045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.782106 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.782328 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.782329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.782329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.782519 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.782573 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.782745 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.782842 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 09:04:54.782939 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.776022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.98.2:44065)
0928 09:04:54.776191 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.776202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.776275 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.776960 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.776961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.777175 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:54.777251 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.777254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.778041 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.778042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.778650 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.778652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.779046 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.779047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.779120 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.779498 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:54.779503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:54.779906 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:54.781530 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.781532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.781532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.781786 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.782173 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.782174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.782175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.782242 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.782593 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:54.782594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:54.782642 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.782720 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.782820 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 09:04:54.802345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34941
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:54.887068 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:54.887264 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:54.887287 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:54.888378 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:54.888413 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:54.888433 8257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 09:04:54.888453 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:54.889861 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:54.890043 8257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:54.890223 8257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:54.895094 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:54.895087 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:54.895087 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:54.895241 8257 server_base.cc:1034] running on GCE node
I20260928 09:04:54.895571 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:54.895815 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:54.896972 8257 hybrid_clock.cc:648] HybridClock initialized: now 1790586294896949 us; error 37 us; skew 500 ppm
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:54.899356 8257 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 09:04:54.899757 8257 webserver.cc:460] Webserver started at http://127.0.98.3:36567/ using document root <none> and password file <none>
I20260928 09:04:54.899992 8257 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:54.900058 8257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:54.900199 8257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:54.901255 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "c92ccf5b146c4a70bbda7cf81a7d0609"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "65eb3ec7d49f92782be4e7cda1707885"
server_key_iv: "5606807a65046973d82b76cf592e8caf"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.901674 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c92ccf5b146c4a70bbda7cf81a7d0609"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "65eb3ec7d49f92782be4e7cda1707885"
server_key_iv: "5606807a65046973d82b76cf592e8caf"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.902976 8257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:54.903676 8272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.903846 8257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:54.903959 8257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "c92ccf5b146c4a70bbda7cf81a7d0609"
format_stamp: "Formatted at 2026-09-28 09:04:54 on dist-test-slave-smns"
server_key: "65eb3ec7d49f92782be4e7cda1707885"
server_key_iv: "5606807a65046973d82b76cf592e8caf"
server_key_version: "encryptionkey@0"
I20260928 09:04:54.904037 8257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:54.914574 8257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:54.916330 8257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:54.916483 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:54.916998 8257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:54.917066 8257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:54.917107 8257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:54.917132 8257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:54.924214 8257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42627
I20260928 09:04:54.924273 8387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42627 every 8 connection(s)
I20260928 09:04:54.924618 8257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:04:54.926875 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.917519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54031 (local address 127.0.98.62:34941)
0928 09:04:54.917599 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.917602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.917771 (+ 169us) server_negotiation.cc:407] Connection header received
0928 09:04:54.918463 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.918464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.918478 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.918560 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.919164 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.920001 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.921058 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.921253 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.921341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.924037 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.924039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.924039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.924053 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.925762 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.926129 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.926129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.926130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.926180 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.926375 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.926376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.926377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.926474 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.926541 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.926746 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.926813 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 09:04:54.927057 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.917475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.98.3:54031)
0928 09:04:54.917645 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.917684 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.918338 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.918643 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.918647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.918895 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:54.919091 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.919096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.920117 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.920118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.920936 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.920938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.921361 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.921361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.921495 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.922056 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:54.922066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:54.923945 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:54.925832 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.925836 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.925838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.926064 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.926234 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.926234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.926235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.926325 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.926543 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:54.926547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:54.926654 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.926733 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.926864 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260928 09:04:54.927269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 8257
I20260928 09:04:54.927347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:54.927544 392 external_mini_cluster.cc:1428] Setting key 4fc114edfeb5b85201cecde78b5a52af
I20260928 09:04:54.927572 8388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34941
I20260928 09:04:54.927654 8388 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:54.927834 8388 heartbeater.cc:511] Master 127.0.98.62:34941 requested a full tablet report, sending...
I20260928 09:04:54.928400 7904 ts_manager.cc:194] Registered new tserver with Master: c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:54.928946 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54031
I20260928 09:04:54.933867 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.928705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40903 (local address 127.0.98.62:34941)
0928 09:04:54.928743 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.928745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.928840 (+ 95us) server_negotiation.cc:407] Connection header received
0928 09:04:54.928888 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.928888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.928899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.928954 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.929285 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.929885 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.930618 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.930814 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.930894 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.931708 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.931709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.931709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.931716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.932769 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.933105 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.933106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.933107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.933165 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.933365 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.933365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.933366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.933474 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.933552 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.933700 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.933773 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:54.933882 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.928623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.98.3:40903)
0928 09:04:54.928771 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.928787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.928836 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.928956 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.928957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.929159 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:54.929208 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.929210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.929980 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.929981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.930514 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.930516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.930893 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.930894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.930955 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.931328 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:54.931332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:54.931619 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:54.932854 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.932855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.932855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.933030 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.933241 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:54.933242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:54.933242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:54.933302 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:54.933559 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:54.933560 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:54.933602 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.933676 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.933774 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260928 09:04:54.939172 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 09:04:54.959661 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:54.954515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.98.62:34941)
0928 09:04:54.954582 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.954583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.954599 (+ 16us) server_negotiation.cc:407] Connection header received
0928 09:04:54.954624 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.954625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.954633 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.954685 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.955259 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.955747 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.956471 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.956641 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.956689 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.959555 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48680: BlockingRecv error: recv got EOF from 127.0.0.1:48680 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 09:04:54 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:54.970906 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.960500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.0.98.62:34941)
0928 09:04:54.960542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.960543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.960598 (+ 55us) server_negotiation.cc:407] Connection header received
0928 09:04:54.960633 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.960634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.960641 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.960684 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:54.961097 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.961636 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.962277 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.962458 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.962506 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.968513 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:54.968514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:54.968515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:54.968521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:54.970033 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.970325 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.970326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.970326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.970386 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:54.970536 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:54.970536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:54.970537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:54.970634 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:54.970702 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.970785 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.970839 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:04:54.972453 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:54.974082 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:54.980624 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.976375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.98.1:34921)
0928 09:04:54.976559 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.976567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.976578 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:04:54.976620 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.976622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.976639 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.976723 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:54.976890 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.978067 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.979364 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.980008 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.980076 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.980156 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.980439 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.980482 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260928 09:04:54.980662 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.976384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.0.98.2:44921)
0928 09:04:54.976844 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.976849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.976858 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:54.976893 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.976895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.976909 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.976980 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:54.977227 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.978281 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.979736 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.980289 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.980377 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.980457 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.980519 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.980557 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":67,"threads_started":1}
I20260928 09:04:54.980855 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.976266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.0.1:32962)
0928 09:04:54.976424 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.976463 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.976524 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.976736 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.976739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.976740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:54.976805 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.976807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.978155 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.978156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.979278 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.979280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.980189 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.980190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.980274 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.980287 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.980353 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.980771 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":30,"thread_start_us":84,"threads_started":1}
I20260928 09:04:54.981038 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.976663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.0.98.3:42627)
0928 09:04:54.976819 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:54.976824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:54.977191 (+ 367us) server_negotiation.cc:407] Connection header received
0928 09:04:54.977233 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:54.977235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:54.977257 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:54.977334 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:54.977446 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.978268 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.979700 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.980482 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.980543 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.980615 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:54.980895 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 09:04:54.980947 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260928 09:04:54.981163 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.976452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.0.1:57556)
0928 09:04:54.977070 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.977079 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.977125 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.977343 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.977344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.977346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:54.977392 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.977394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.978369 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.978369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.979611 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.979614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.980649 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.980650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.980730 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.980734 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.980789 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.980820 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":35,"thread_start_us":96,"threads_started":1}
I20260928 09:04:54.981791 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:54.976266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.0.1:42260)
0928 09:04:54.976539 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:54.976550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:54.976605 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:54.976998 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:54.976999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:54.977001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:54.977048 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.977110 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.978383 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.978384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.979676 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:54.979678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:54.980376 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:54.980377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:54.980437 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:54.980442 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:04:54.980503 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:04:54.981740 (+ 1237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1248,"thread_start_us":54,"threads_started":1}
I20260928 09:04:54.982021 8042 tablet_service.cc:1467] Processing CreateTablet for tablet ae0bd81c0a34426b8a3d879f92d136fb (DEFAULT_TABLE table=test-table [id=9f64c84718824b18830158221cdd3ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:54.982074 8322 tablet_service.cc:1467] Processing CreateTablet for tablet ae0bd81c0a34426b8a3d879f92d136fb (DEFAULT_TABLE table=test-table [id=9f64c84718824b18830158221cdd3ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:54.982405 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0bd81c0a34426b8a3d879f92d136fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:54.982405 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0bd81c0a34426b8a3d879f92d136fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:54.982540 8182 tablet_service.cc:1467] Processing CreateTablet for tablet ae0bd81c0a34426b8a3d879f92d136fb (DEFAULT_TABLE table=test-table [id=9f64c84718824b18830158221cdd3ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:54.982864 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0bd81c0a34426b8a3d879f92d136fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:54.986020 8416 tablet_bootstrap.cc:492] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Bootstrap starting.
I20260928 09:04:54.986451 8417 tablet_bootstrap.cc:492] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Bootstrap starting.
I20260928 09:04:54.986509 8418 tablet_bootstrap.cc:492] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Bootstrap starting.
I20260928 09:04:54.986811 8416 tablet_bootstrap.cc:654] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:54.987087 8418 tablet_bootstrap.cc:654] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:54.987130 8417 tablet_bootstrap.cc:654] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:54.987157 8416 log.cc:824] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:54.987421 8418 log.cc:824] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:54.987449 8417 log.cc:824] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:54.988162 8417 tablet_bootstrap.cc:492] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: No bootstrap required, opened a new log
I20260928 09:04:54.988162 8416 tablet_bootstrap.cc:492] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: No bootstrap required, opened a new log
I20260928 09:04:54.988237 8416 ts_tablet_manager.cc:1397] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:54.988237 8417 ts_tablet_manager.cc:1397] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:54.988296 8418 tablet_bootstrap.cc:492] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: No bootstrap required, opened a new log
I20260928 09:04:54.988364 8418 ts_tablet_manager.cc:1397] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:54.989823 8418 raft_consensus.cc:348] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:54.989980 8418 raft_consensus.cc:374] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:54.990024 8418 raft_consensus.cc:729] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d256718298d347659b4a8bcfd60974e7, State: Initialized, Role: FOLLOWER
I20260928 09:04:54.990163 8418 consensus_queue.cc:260] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:54.990258 8416 raft_consensus.cc:348] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:54.990259 8417 raft_consensus.cc:348] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:54.990423 8416 raft_consensus.cc:374] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:54.990433 8417 raft_consensus.cc:374] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:54.990450 8418 ts_tablet_manager.cc:1428] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:04:54.990470 8416 raft_consensus.cc:729] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Initialized, Role: FOLLOWER
I20260928 09:04:54.990470 8417 raft_consensus.cc:729] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Initialized, Role: FOLLOWER
I20260928 09:04:54.990592 8108 heartbeater.cc:503] Master 127.0.98.62:34941 was elected leader, sending a full tablet report...
I20260928 09:04:54.990588 8416 consensus_queue.cc:260] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:54.990588 8417 consensus_queue.cc:260] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:54.990847 8417 ts_tablet_manager.cc:1428] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:04:54.990847 8416 ts_tablet_manager.cc:1428] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:04:54.990876 8248 heartbeater.cc:503] Master 127.0.98.62:34941 was elected leader, sending a full tablet report...
I20260928 09:04:54.990957 8388 heartbeater.cc:503] Master 127.0.98.62:34941 was elected leader, sending a full tablet report...
W20260928 09:04:55.023322 8249 tablet.cc:2369] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:04:55.104629 8109 tablet.cc:2369] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:55.109174 8424 raft_consensus.cc:482] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:55.109287 8424 raft_consensus.cc:504] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.109767 8424 leader_election.cc:288] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921), d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921)
I20260928 09:04:55.112771 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.110067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36341 (local address 127.0.98.1:34921)
0928 09:04:55.110142 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.110145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.110339 (+ 194us) server_negotiation.cc:407] Connection header received
0928 09:04:55.110389 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.110391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.110404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.110465 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:55.110642 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.111099 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.112074 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.112423 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.112479 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.112494 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.112661 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.112705 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 09:04:55.112774 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.109993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.98.3:36341)
0928 09:04:55.110263 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.110273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.110330 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.110495 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.110496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.110498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:55.110574 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.110575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.111239 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.111240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.111981 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.111983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.112504 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.112505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.112578 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.112582 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.112634 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.112662 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":116,"thread_start_us":99,"threads_started":1}
I20260928 09:04:55.112845 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.110120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60953 (local address 127.0.98.2:44921)
0928 09:04:55.110177 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.110179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.110216 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:04:55.110248 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.110249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.110261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.110322 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:55.110509 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.111045 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.112083 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.112482 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.112549 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.112559 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.112729 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.112768 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:04:55.112900 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.109994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.98.3:60953)
0928 09:04:55.110083 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.110096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.110156 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.110353 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.110353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.110356 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:55.110435 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.110438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.111124 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.111125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.111970 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.111972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.112567 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.112567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.112650 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.112654 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.112711 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.112749 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260928 09:04:55.113168 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0bd81c0a34426b8a3d879f92d136fb" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7" is_pre_election: true
I20260928 09:04:55.113322 8062 raft_consensus.cc:2454] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 0.
I20260928 09:04:55.113349 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0bd81c0a34426b8a3d879f92d136fb" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9168cd27a9458cafdde6f28beedd6e" is_pre_election: true
I20260928 09:04:55.113509 8201 raft_consensus.cc:2454] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 0.
I20260928 09:04:55.113574 8276 leader_election.cc:302] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92ccf5b146c4a70bbda7cf81a7d0609, d256718298d347659b4a8bcfd60974e7; no voters:
I20260928 09:04:55.113703 8424 raft_consensus.cc:2781] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:55.113751 8424 raft_consensus.cc:482] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:55.113775 8424 raft_consensus.cc:3037] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.114713 8424 raft_consensus.cc:504] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.114858 8424 leader_election.cc:288] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 election: Requested vote from peers ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921), d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921)
I20260928 09:04:55.115084 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0bd81c0a34426b8a3d879f92d136fb" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9168cd27a9458cafdde6f28beedd6e"
I20260928 09:04:55.115110 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0bd81c0a34426b8a3d879f92d136fb" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7"
I20260928 09:04:55.115168 8201 raft_consensus.cc:3037] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.115175 8062 raft_consensus.cc:3037] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.116025 8201 raft_consensus.cc:2454] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 1.
I20260928 09:04:55.116025 8062 raft_consensus.cc:2454] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 1.
I20260928 09:04:55.116245 8273 leader_election.cc:302] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, c92ccf5b146c4a70bbda7cf81a7d0609; no voters:
I20260928 09:04:55.116349 8424 raft_consensus.cc:2781] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:55.116526 8424 raft_consensus.cc:686] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 LEADER]: Becoming Leader. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Running, Role: LEADER
I20260928 09:04:55.116627 8424 consensus_queue.cc:237] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.117401 7904 catalog_manager.cc:5495] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: HEALTHY } } }
I20260928 09:04:55.124011 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.121680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.0.98.3:42627)
0928 09:04:55.121719 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.121721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.121740 (+ 19us) server_negotiation.cc:407] Connection header received
0928 09:04:55.121824 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.121825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.121840 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.121891 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.122023 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.122447 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.123261 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.123432 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.123481 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.123635 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.123728 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.123787 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.123938 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.123955 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:55.125758 8201 raft_consensus.cc:1261] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Refusing update from remote peer c92ccf5b146c4a70bbda7cf81a7d0609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.125771 8062 raft_consensus.cc:1261] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Refusing update from remote peer c92ccf5b146c4a70bbda7cf81a7d0609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.126067 8424 consensus_queue.cc:1035] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.126214 8426 consensus_queue.cc:1035] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.129134 8430 mvcc.cc:205] Tried to move back new op lower bound from 7334241464832495616 to 7334241464798412800. Current Snapshot: MvccSnapshot[applied={T|T < 7334241464832495616 or (T in {7334241464832495616})}]
I20260928 09:04:55.131134 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.131232 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.131940 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Sending DeleteTablet for 3 replicas of tablet ae0bd81c0a34426b8a3d879f92d136fb
I20260928 09:04:55.132236 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0bd81c0a34426b8a3d879f92d136fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42260
I20260928 09:04:55.132319 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0bd81c0a34426b8a3d879f92d136fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260928 09:04:55.132421 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0bd81c0a34426b8a3d879f92d136fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57556
I20260928 09:04:55.132403 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:55.132537 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:55.132647 8440 tablet_replica.cc:330] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: stopping tablet replica
I20260928 09:04:55.132745 8440 raft_consensus.cc:2229] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:55.132858 8440 raft_consensus.cc:2258] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.132961 8438 tablet_replica.cc:330] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: stopping tablet replica
I20260928 09:04:55.133092 8438 raft_consensus.cc:2229] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.133319 8440 ts_tablet_manager.cc:1905] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.133545 8438 raft_consensus.cc:2258] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.133848 8182 tablet_service.cc:1467] Processing CreateTablet for tablet d897d618f487444eb60376797451c3fa (DEFAULT_TABLE table=test-table [id=521041c06daa4b0ea684b99d0e4c4023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.133885 8322 tablet_service.cc:1467] Processing CreateTablet for tablet d897d618f487444eb60376797451c3fa (DEFAULT_TABLE table=test-table [id=521041c06daa4b0ea684b99d0e4c4023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.133975 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897d618f487444eb60376797451c3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.134003 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897d618f487444eb60376797451c3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.134230 8439 tablet_replica.cc:330] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: stopping tablet replica
I20260928 09:04:55.134315 8439 raft_consensus.cc:2229] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.134399 8439 raft_consensus.cc:2258] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.134835 8439 ts_tablet_manager.cc:1905] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.135372 8416 tablet_bootstrap.cc:492] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Bootstrap starting.
I20260928 09:04:55.135423 8042 tablet_service.cc:1467] Processing CreateTablet for tablet d897d618f487444eb60376797451c3fa (DEFAULT_TABLE table=test-table [id=521041c06daa4b0ea684b99d0e4c4023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.135525 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897d618f487444eb60376797451c3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.135845 8416 tablet_bootstrap.cc:654] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.135869 8417 tablet_bootstrap.cc:492] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Bootstrap starting.
I20260928 09:04:55.136291 8417 tablet_bootstrap.cc:654] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.137066 8440 ts_tablet_manager.cc:1918] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:55.137068 8439 ts_tablet_manager.cc:1918] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:55.137137 8439 log.cc:1196] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/ae0bd81c0a34426b8a3d879f92d136fb
I20260928 09:04:55.137137 8440 log.cc:1196] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/ae0bd81c0a34426b8a3d879f92d136fb
I20260928 09:04:55.137388 8439 ts_tablet_manager.cc:1939] T ae0bd81c0a34426b8a3d879f92d136fb P d256718298d347659b4a8bcfd60974e7: Deleting consensus metadata
I20260928 09:04:55.137389 8440 ts_tablet_manager.cc:1939] T ae0bd81c0a34426b8a3d879f92d136fb P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting consensus metadata
I20260928 09:04:55.137809 7892 catalog_manager.cc:4841] TS c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627): tablet ae0bd81c0a34426b8a3d879f92d136fb (table test-table [id=9f64c84718824b18830158221cdd3ea6]) successfully deleted
I20260928 09:04:55.138021 8438 ts_tablet_manager.cc:1905] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.138969 8418 tablet_bootstrap.cc:492] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Bootstrap starting.
I20260928 09:04:55.139367 8418 tablet_bootstrap.cc:654] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.139424 7892 catalog_manager.cc:4841] TS d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921): tablet ae0bd81c0a34426b8a3d879f92d136fb (table test-table [id=9f64c84718824b18830158221cdd3ea6]) successfully deleted
I20260928 09:04:55.139730 8438 ts_tablet_manager.cc:1918] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:55.139842 8438 log.cc:1196] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/ae0bd81c0a34426b8a3d879f92d136fb
I20260928 09:04:55.139955 8418 tablet_bootstrap.cc:492] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: No bootstrap required, opened a new log
I20260928 09:04:55.139999 8418 ts_tablet_manager.cc:1397] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:55.140112 8438 ts_tablet_manager.cc:1939] T ae0bd81c0a34426b8a3d879f92d136fb P ae9168cd27a9458cafdde6f28beedd6e: Deleting consensus metadata
I20260928 09:04:55.140125 8418 raft_consensus.cc:348] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.140221 8418 raft_consensus.cc:374] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.140249 8418 raft_consensus.cc:729] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d256718298d347659b4a8bcfd60974e7, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.140290 8418 consensus_queue.cc:260] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.140513 7889 catalog_manager.cc:4841] TS ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921): tablet ae0bd81c0a34426b8a3d879f92d136fb (table test-table [id=9f64c84718824b18830158221cdd3ea6]) successfully deleted
I20260928 09:04:55.140617 8418 ts_tablet_manager.cc:1428] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:55.140722 8417 tablet_bootstrap.cc:492] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: No bootstrap required, opened a new log
I20260928 09:04:55.140774 8417 ts_tablet_manager.cc:1397] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 09:04:55.140936 8417 raft_consensus.cc:348] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.141010 8417 raft_consensus.cc:374] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.141031 8417 raft_consensus.cc:729] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.141072 8417 consensus_queue.cc:260] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.141167 8417 ts_tablet_manager.cc:1428] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.141534 8416 tablet_bootstrap.cc:492] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: No bootstrap required, opened a new log
I20260928 09:04:55.141604 8416 ts_tablet_manager.cc:1397] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 09:04:55.141733 8416 raft_consensus.cc:348] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.141798 8416 raft_consensus.cc:374] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.141819 8416 raft_consensus.cc:729] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.141857 8416 consensus_queue.cc:260] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.141960 8416 ts_tablet_manager.cc:1428] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 09:04:55.175182 8389 tablet.cc:2369] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:04:55.277200 8422 raft_consensus.cc:482] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:55.277315 8422 raft_consensus.cc:504] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.277756 8422 leader_election.cc:288] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921), c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:55.280936 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.277996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60337 (local address 127.0.98.2:44921)
0928 09:04:55.278045 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.278048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.278205 (+ 157us) server_negotiation.cc:407] Connection header received
0928 09:04:55.278278 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.278279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.278287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.278355 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:55.278529 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.278982 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.280218 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.280563 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.280645 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.280657 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.280826 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.280861 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:04:55.280997 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.277943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.98.1:51305)
0928 09:04:55.278073 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.278085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.278153 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.278307 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.278308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.278310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:55.278383 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.278385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.279019 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.279020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.280115 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.280118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.280726 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.280727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.280801 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.280805 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.280874 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.280913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 09:04:55.280979 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.278055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51305 (local address 127.0.98.3:42627)
0928 09:04:55.278107 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.278109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.278138 (+ 29us) server_negotiation.cc:407] Connection header received
0928 09:04:55.278233 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.278234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.278243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.278301 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:55.278456 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.278925 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.280202 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.280639 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.280725 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.280734 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.280879 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.280914 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:04:55.281185 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.277975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.98.1:60337)
0928 09:04:55.278123 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.278133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.278212 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.278372 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.278373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.278375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:55.278452 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.278454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.279105 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.279106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.280103 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.280105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.280662 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.280664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.280741 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.280745 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.280818 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.280855 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":13,"thread_start_us":103,"threads_started":1}
I20260928 09:04:55.281230 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897d618f487444eb60376797451c3fa" candidate_uuid: "d256718298d347659b4a8bcfd60974e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" is_pre_election: true
I20260928 09:04:55.281375 8342 raft_consensus.cc:2454] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d256718298d347659b4a8bcfd60974e7 in term 0.
I20260928 09:04:55.281438 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897d618f487444eb60376797451c3fa" candidate_uuid: "d256718298d347659b4a8bcfd60974e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9168cd27a9458cafdde6f28beedd6e" is_pre_election: true
I20260928 09:04:55.281524 8201 raft_consensus.cc:2454] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d256718298d347659b4a8bcfd60974e7 in term 0.
I20260928 09:04:55.281587 7996 leader_election.cc:302] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92ccf5b146c4a70bbda7cf81a7d0609, d256718298d347659b4a8bcfd60974e7; no voters:
I20260928 09:04:55.281754 8422 raft_consensus.cc:2781] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:55.281819 8422 raft_consensus.cc:482] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:55.281858 8422 raft_consensus.cc:3037] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.282595 8422 raft_consensus.cc:504] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.282737 8422 leader_election.cc:288] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [CANDIDATE]: Term 1 election: Requested vote from peers ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921), c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:55.282892 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897d618f487444eb60376797451c3fa" candidate_uuid: "d256718298d347659b4a8bcfd60974e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9168cd27a9458cafdde6f28beedd6e"
I20260928 09:04:55.282967 8201 raft_consensus.cc:3037] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.282994 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897d618f487444eb60376797451c3fa" candidate_uuid: "d256718298d347659b4a8bcfd60974e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609"
I20260928 09:04:55.283082 8342 raft_consensus.cc:3037] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.283581 8201 raft_consensus.cc:2454] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d256718298d347659b4a8bcfd60974e7 in term 1.
I20260928 09:04:55.283746 7993 leader_election.cc:302] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, d256718298d347659b4a8bcfd60974e7; no voters:
I20260928 09:04:55.283820 8342 raft_consensus.cc:2454] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d256718298d347659b4a8bcfd60974e7 in term 1.
I20260928 09:04:55.283844 8422 raft_consensus.cc:2781] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:55.283972 8422 raft_consensus.cc:686] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 1 LEADER]: Becoming Leader. State: Replica: d256718298d347659b4a8bcfd60974e7, State: Running, Role: LEADER
I20260928 09:04:55.284044 8422 consensus_queue.cc:237] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.284739 7902 catalog_manager.cc:5495] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d256718298d347659b4a8bcfd60974e7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d256718298d347659b4a8bcfd60974e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: UNKNOWN } } }
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:55.345156 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.338320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34941 (local address 127.0.0.1:48700)
0928 09:04:55.338514 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.338525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.338592 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.338737 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.338738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.339044 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:55.339093 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.339094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.339712 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.339712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.340232 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.340233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.340536 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.340537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.340574 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.341054 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:55.341058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:55.342493 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:55.344158 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:55.344164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:55.344166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:55.344348 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:55.344511 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:55.344512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:55.344512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:55.344566 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:55.344767 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:55.344774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:55.344821 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.344907 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.345029 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":112,"threads_started":1}
I20260928 09:04:55.345163 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.338331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.0.98.62:34941)
0928 09:04:55.338401 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.338404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.338618 (+ 214us) server_negotiation.cc:407] Connection header received
0928 09:04:55.338647 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.338647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.338657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.338714 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:55.339163 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.339648 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.340299 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.340476 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.340524 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.342583 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:55.342584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:55.342584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:55.342589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:55.344094 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:55.344398 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:55.344399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:55.344399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:55.344451 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:55.344604 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:55.344604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:55.344605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:55.344693 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:55.344776 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.344974 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.345086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":40}
I20260928 09:04:55.347342 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 09:04:55.347579 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:55.348968 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 526b6b07b061409e8a00ddda5ef90de7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d30a64e687f24ebd897d82ceb12e87ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:55.348977 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 526b6b07b061409e8a00ddda5ef90de7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d30a64e687f24ebd897d82ceb12e87ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:55.349051 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 526b6b07b061409e8a00ddda5ef90de7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d30a64e687f24ebd897d82ceb12e87ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 09:04:55.349126 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526b6b07b061409e8a00ddda5ef90de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.349133 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526b6b07b061409e8a00ddda5ef90de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.349174 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526b6b07b061409e8a00ddda5ef90de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.350430 8416 tablet_bootstrap.cc:492] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: Bootstrap starting.
I20260928 09:04:55.350616 8418 tablet_bootstrap.cc:492] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: Bootstrap starting.
I20260928 09:04:55.350863 8416 tablet_bootstrap.cc:654] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.350981 8418 tablet_bootstrap.cc:654] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.351321 8417 tablet_bootstrap.cc:492] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: Bootstrap starting.
I20260928 09:04:55.351457 8416 tablet_bootstrap.cc:492] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: No bootstrap required, opened a new log
I20260928 09:04:55.351507 8416 ts_tablet_manager.cc:1397] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:55.351625 8418 tablet_bootstrap.cc:492] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: No bootstrap required, opened a new log
I20260928 09:04:55.351646 8417 tablet_bootstrap.cc:654] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.351630 8416 raft_consensus.cc:348] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.351677 8416 raft_consensus.cc:374] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.351670 8418 ts_tablet_manager.cc:1397] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:55.351691 8416 raft_consensus.cc:729] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.351713 8416 consensus_queue.cc:260] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.351800 8416 ts_tablet_manager.cc:1428] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.351816 8418 raft_consensus.cc:348] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.351894 8418 raft_consensus.cc:374] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.351861 8423 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.351920 8418 raft_consensus.cc:729] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d256718298d347659b4a8bcfd60974e7, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.351948 8423 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.351965 8418 consensus_queue.cc:260] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.352078 8418 ts_tablet_manager.cc:1428] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.352118 8422 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.352188 8422 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.352439 8417 tablet_bootstrap.cc:492] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: No bootstrap required, opened a new log
I20260928 09:04:55.352506 8417 ts_tablet_manager.cc:1397] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:55.352635 8417 raft_consensus.cc:348] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.352703 8417 raft_consensus.cc:374] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.352734 8417 raft_consensus.cc:729] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.352785 8417 consensus_queue.cc:260] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.352888 8417 ts_tablet_manager.cc:1428] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.352918 8424 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.353024 8424 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.487902 8424 raft_consensus.cc:482] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:55.487998 8424 raft_consensus.cc:504] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.488152 8424 leader_election.cc:288] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921), ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921)
I20260928 09:04:55.488399 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526b6b07b061409e8a00ddda5ef90de7" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7" is_pre_election: true
I20260928 09:04:55.488454 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526b6b07b061409e8a00ddda5ef90de7" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9168cd27a9458cafdde6f28beedd6e" is_pre_election: true
I20260928 09:04:55.488526 8062 raft_consensus.cc:2454] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 0.
I20260928 09:04:55.488531 8201 raft_consensus.cc:2454] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 0.
I20260928 09:04:55.488706 8273 leader_election.cc:302] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, c92ccf5b146c4a70bbda7cf81a7d0609; no voters:
I20260928 09:04:55.488852 8424 raft_consensus.cc:2781] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:55.488907 8424 raft_consensus.cc:482] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:55.488924 8424 raft_consensus.cc:3037] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.489676 8424 raft_consensus.cc:504] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.489813 8424 leader_election.cc:288] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 election: Requested vote from peers d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921), ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921)
I20260928 09:04:55.489967 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526b6b07b061409e8a00ddda5ef90de7" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9168cd27a9458cafdde6f28beedd6e"
I20260928 09:04:55.489979 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526b6b07b061409e8a00ddda5ef90de7" candidate_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7"
I20260928 09:04:55.490046 8201 raft_consensus.cc:3037] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.490063 8062 raft_consensus.cc:3037] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.490651 8201 raft_consensus.cc:2454] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 1.
I20260928 09:04:55.490724 8062 raft_consensus.cc:2454] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92ccf5b146c4a70bbda7cf81a7d0609 in term 1.
I20260928 09:04:55.490818 8273 leader_election.cc:302] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, c92ccf5b146c4a70bbda7cf81a7d0609; no voters:
I20260928 09:04:55.490909 8424 raft_consensus.cc:2781] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:55.490974 8424 raft_consensus.cc:686] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 LEADER]: Becoming Leader. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Running, Role: LEADER
I20260928 09:04:55.491032 8424 consensus_queue.cc:237] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.491285 8426 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: TxnStatusTablet state changed. Reason: New leader c92ccf5b146c4a70bbda7cf81a7d0609. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.491401 8426 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:55.491554 8458 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 09:04:55.491663 8458 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 09:04:55.491662 7902 catalog_manager.cc:5495] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: HEALTHY } } }
I20260928 09:04:55.497710 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.495176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.98.3:42627)
0928 09:04:55.495217 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.495219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.495260 (+ 41us) server_negotiation.cc:407] Connection header received
0928 09:04:55.495343 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.495344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.495353 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.495397 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.495510 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.495964 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.497018 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.497215 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.497266 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.497467 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.497493 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.497534 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.497641 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.497659 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:04:55.497759 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.495120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.0.1:57586)
0928 09:04:55.495206 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.495218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.495271 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.495412 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.495413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.495415 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:55.495463 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.495465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.496035 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.496036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.496934 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.496937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.497290 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.497291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.497367 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.497383 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.497574 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.497583 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.497634 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.497682 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 09:04:55.498652 8201 raft_consensus.cc:1261] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Refusing update from remote peer c92ccf5b146c4a70bbda7cf81a7d0609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.498682 8062 raft_consensus.cc:1261] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Refusing update from remote peer c92ccf5b146c4a70bbda7cf81a7d0609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.498891 8426 consensus_queue.cc:1035] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.499018 8424 consensus_queue.cc:1035] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.499509 8422 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: TxnStatusTablet state changed. Reason: New leader c92ccf5b146c4a70bbda7cf81a7d0609. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.499614 8422 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.499581 8423 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: TxnStatusTablet state changed. Reason: New leader c92ccf5b146c4a70bbda7cf81a7d0609. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.499661 8423 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.500128 8426 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.500211 8426 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:55.500308 8431 mvcc.cc:205] Tried to move back new op lower bound from 7334241466360893440 to 7334241466331672576. Current Snapshot: MvccSnapshot[applied={T|T < 7334241466360893440 or (T in {7334241466360893440})}]
I20260928 09:04:55.500535 8424 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.500600 8424 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P c92ccf5b146c4a70bbda7cf81a7d0609: This TxnStatusTablet replica's current role is: LEADER
I20260928 09:04:55.500789 8423 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.500855 8423 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P ae9168cd27a9458cafdde6f28beedd6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.500835 8422 tablet_replica.cc:439] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } }
I20260928 09:04:55.500916 8422 tablet_replica.cc:442] T 526b6b07b061409e8a00ddda5ef90de7 P d256718298d347659b4a8bcfd60974e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 09:04:55.504530 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.502107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.0.98.1:34921)
0928 09:04:55.502142 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.502143 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.502152 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:55.502218 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.502219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.502236 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.502302 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.502422 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.502862 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.503864 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.504039 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.504088 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.504239 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.504327 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.504375 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.504439 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.504453 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:04:55.509403 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.506858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40345 (local address 127.0.98.3:42627)
0928 09:04:55.506895 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.506897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.507017 (+ 120us) server_negotiation.cc:407] Connection header received
0928 09:04:55.507064 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.507065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.507074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.507150 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.507259 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.508001 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.508703 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.509001 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.509070 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.509197 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.509226 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.509276 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.509318 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.509337 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:55.509413 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.506791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.98.1:40345)
0928 09:04:55.506940 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.506951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.507013 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.507160 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.507161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.507162 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:55.507216 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.507217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.508070 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.508070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.508632 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.508633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.509073 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.509074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.509116 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.509127 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.509281 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.509282 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.509315 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.509346 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260928 09:04:55.511579 8201 raft_consensus.cc:1261] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Refusing update from remote peer d256718298d347659b4a8bcfd60974e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.511677 8342 raft_consensus.cc:1261] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Refusing update from remote peer d256718298d347659b4a8bcfd60974e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.511844 8446 consensus_queue.cc:1035] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.511932 8422 consensus_queue.cc:1035] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.519812 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.517408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41131 (local address 127.0.98.1:34921)
0928 09:04:55.517508 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.517510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.517603 (+ 93us) server_negotiation.cc:407] Connection header received
0928 09:04:55.517686 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.517687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.517699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.517727 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.517895 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.518335 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.519066 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.519313 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.519371 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.519555 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.519586 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.519614 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.519731 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.519749 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 09:04:55.519843 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.517377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.98.3:41131)
0928 09:04:55.517512 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.517523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.517611 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.517781 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.517782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.517785 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:55.517823 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.517824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.518414 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.518415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.518976 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.518977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.519369 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.519369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.519425 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.519437 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.519657 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.519658 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.519723 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.519769 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260928 09:04:55.529644 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.529749 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.530347 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Sending DeleteTablet for 3 replicas of tablet d897d618f487444eb60376797451c3fa
I20260928 09:04:55.530681 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet d897d618f487444eb60376797451c3fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260928 09:04:55.530769 8439 tablet_replica.cc:330] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: stopping tablet replica
I20260928 09:04:55.530808 8439 raft_consensus.cc:2229] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:55.530884 8439 raft_consensus.cc:2258] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.531056 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet d897d618f487444eb60376797451c3fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57556
I20260928 09:04:55.531177 8440 tablet_replica.cc:330] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: stopping tablet replica
I20260928 09:04:55.531248 8440 raft_consensus.cc:2229] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.531282 8440 raft_consensus.cc:2258] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.531585 8440 ts_tablet_manager.cc:1905] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.531636 8439 ts_tablet_manager.cc:1905] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.533327 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet d897d618f487444eb60376797451c3fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42260
I20260928 09:04:55.533414 8438 tablet_replica.cc:330] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: stopping tablet replica
I20260928 09:04:55.533456 8438 raft_consensus.cc:2229] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.533531 8438 raft_consensus.cc:2258] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.533620 8440 ts_tablet_manager.cc:1918] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:55.533672 8440 log.cc:1196] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/d897d618f487444eb60376797451c3fa
I20260928 09:04:55.533892 8440 ts_tablet_manager.cc:1939] T d897d618f487444eb60376797451c3fa P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting consensus metadata
I20260928 09:04:55.534214 7892 catalog_manager.cc:4841] TS c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627): tablet d897d618f487444eb60376797451c3fa (table test-table [id=521041c06daa4b0ea684b99d0e4c4023]) successfully deleted
I20260928 09:04:55.536825 8439 ts_tablet_manager.cc:1918] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:55.536878 8439 log.cc:1196] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/d897d618f487444eb60376797451c3fa
I20260928 09:04:55.537113 8439 ts_tablet_manager.cc:1939] T d897d618f487444eb60376797451c3fa P d256718298d347659b4a8bcfd60974e7: Deleting consensus metadata
I20260928 09:04:55.537413 7892 catalog_manager.cc:4841] TS d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921): tablet d897d618f487444eb60376797451c3fa (table test-table [id=521041c06daa4b0ea684b99d0e4c4023]) successfully deleted
I20260928 09:04:55.537743 8438 ts_tablet_manager.cc:1905] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.539346 8438 ts_tablet_manager.cc:1918] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:55.539399 8438 log.cc:1196] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/d897d618f487444eb60376797451c3fa
I20260928 09:04:55.539626 8438 ts_tablet_manager.cc:1939] T d897d618f487444eb60376797451c3fa P ae9168cd27a9458cafdde6f28beedd6e: Deleting consensus metadata
I20260928 09:04:55.540146 7889 catalog_manager.cc:4841] TS ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921): tablet d897d618f487444eb60376797451c3fa (table test-table [id=521041c06daa4b0ea684b99d0e4c4023]) successfully deleted
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 09:04:55.557013 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:55.551667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.0.98.62:34941)
0928 09:04:55.551717 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.551719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.551849 (+ 130us) server_negotiation.cc:407] Connection header received
0928 09:04:55.551966 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.551967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.551977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.552042 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:55.552607 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.553099 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.553921 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.554115 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.554203 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.556934 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48714: BlockingRecv error: recv got EOF from 127.0.0.1:48714 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 09:04:55.564854 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.557839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.0.98.62:34941)
0928 09:04:55.557880 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.557882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.558044 (+ 162us) server_negotiation.cc:407] Connection header received
0928 09:04:55.558114 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.558115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.558125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.558199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:55.558636 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.559095 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.559779 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.559965 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.560063 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.562221 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:55.562245 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:55.562246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:55.562252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:55.563693 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:55.564070 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:55.564071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:55.564072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:55.564132 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:55.564332 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:55.564333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:55.564334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:55.564451 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:55.564565 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.564669 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.564745 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:04:55.566581 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:04:55.566772 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:55.568121 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3578cd01d747f987c45a81116a5325 (DEFAULT_TABLE table=test-table [id=79f1546836ea455da4b49a84775fcba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.568121 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3578cd01d747f987c45a81116a5325 (DEFAULT_TABLE table=test-table [id=79f1546836ea455da4b49a84775fcba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.568251 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3578cd01d747f987c45a81116a5325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.568251 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3578cd01d747f987c45a81116a5325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.569598 8418 tablet_bootstrap.cc:492] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Bootstrap starting.
I20260928 09:04:55.569599 8417 tablet_bootstrap.cc:492] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Bootstrap starting.
I20260928 09:04:55.569924 8417 tablet_bootstrap.cc:654] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.570009 8418 tablet_bootstrap.cc:654] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.570794 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3578cd01d747f987c45a81116a5325 (DEFAULT_TABLE table=test-table [id=79f1546836ea455da4b49a84775fcba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.570938 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3578cd01d747f987c45a81116a5325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.571270 8417 tablet_bootstrap.cc:492] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: No bootstrap required, opened a new log
I20260928 09:04:55.571316 8417 ts_tablet_manager.cc:1397] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:04:55.571414 8417 raft_consensus.cc:348] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.571487 8417 raft_consensus.cc:374] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.571506 8417 raft_consensus.cc:729] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.571553 8417 consensus_queue.cc:260] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.571643 8417 ts_tablet_manager.cc:1428] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.572292 8416 tablet_bootstrap.cc:492] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Bootstrap starting.
I20260928 09:04:55.572726 8416 tablet_bootstrap.cc:654] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.573050 8418 tablet_bootstrap.cc:492] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: No bootstrap required, opened a new log
I20260928 09:04:55.573099 8418 ts_tablet_manager.cc:1397] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 09:04:55.573246 8418 raft_consensus.cc:348] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.573324 8418 raft_consensus.cc:374] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.573348 8418 raft_consensus.cc:729] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d256718298d347659b4a8bcfd60974e7, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.573405 8418 consensus_queue.cc:260] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.573475 8416 tablet_bootstrap.cc:492] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: No bootstrap required, opened a new log
I20260928 09:04:55.573513 8416 ts_tablet_manager.cc:1397] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:55.573510 8418 ts_tablet_manager.cc:1428] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.573712 8416 raft_consensus.cc:348] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.573781 8416 raft_consensus.cc:374] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.573808 8416 raft_consensus.cc:729] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.573851 8416 consensus_queue.cc:260] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.573959 8416 ts_tablet_manager.cc:1428] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.584452 8423 raft_consensus.cc:482] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:55.584546 8423 raft_consensus.cc:504] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.584908 8423 leader_election.cc:288] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921), c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:55.589254 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.585123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47325 (local address 127.0.98.1:34921)
0928 09:04:55.585166 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.585168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.585271 (+ 103us) server_negotiation.cc:407] Connection header received
0928 09:04:55.585380 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.585381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.585391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.585433 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:55.585921 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.586673 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.588282 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.588839 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.588884 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.588894 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.589138 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.589183 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:55.589509 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.585065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.98.2:47325)
0928 09:04:55.585214 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.585224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.585299 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.585503 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.585505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.585507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:55.585831 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.585834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.586787 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.586788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.588173 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.588176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.588981 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.588982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.589052 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.589056 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.589366 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.589433 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 09:04:55.589761 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3578cd01d747f987c45a81116a5325" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7" is_pre_election: true
I20260928 09:04:55.589849 8062 raft_consensus.cc:2454] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 0.
I20260928 09:04:55.589838 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.585383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47649 (local address 127.0.98.3:42627)
0928 09:04:55.585423 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.585424 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.585635 (+ 211us) server_negotiation.cc:407] Connection header received
0928 09:04:55.585736 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.585737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.585747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.585775 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:04:55.585999 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.586673 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.588327 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.588840 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.588884 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.588894 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.589219 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.589287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:55.590023 8136 leader_election.cc:302] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, d256718298d347659b4a8bcfd60974e7; no voters:
I20260928 09:04:55.590147 8423 raft_consensus.cc:2781] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:55.590183 8423 raft_consensus.cc:482] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:55.590201 8423 raft_consensus.cc:3037] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.590437 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.585262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.98.2:47649)
0928 09:04:55.585578 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.585589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.585658 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.585834 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.585835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.585837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:04:55.585891 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.585892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.586787 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.586788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.588239 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.588241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.589015 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.589016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.589109 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.589113 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.590310 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.590359 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":25,"thread_start_us":43,"threads_started":1}
I20260928 09:04:55.590755 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3578cd01d747f987c45a81116a5325" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" is_pre_election: true
I20260928 09:04:55.590835 8342 raft_consensus.cc:2454] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 0.
I20260928 09:04:55.591151 8423 raft_consensus.cc:504] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.591293 8423 leader_election.cc:288] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 election: Requested vote from peers d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921), c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:55.591522 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3578cd01d747f987c45a81116a5325" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609"
I20260928 09:04:55.591598 8342 raft_consensus.cc:3037] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.592206 8342 raft_consensus.cc:2454] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 1.
I20260928 09:04:55.592391 8136 leader_election.cc:302] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, c92ccf5b146c4a70bbda7cf81a7d0609; no voters:
I20260928 09:04:55.592466 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3578cd01d747f987c45a81116a5325" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7"
I20260928 09:04:55.592532 8062 raft_consensus.cc:3037] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.593117 8062 raft_consensus.cc:2454] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 1.
I20260928 09:04:55.593279 8423 raft_consensus.cc:2781] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:55.593465 8423 raft_consensus.cc:686] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 1 LEADER]: Becoming Leader. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Running, Role: LEADER
I20260928 09:04:55.593580 8423 consensus_queue.cc:237] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.594219 7904 catalog_manager.cc:5495] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ae9168cd27a9458cafdde6f28beedd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:55.602366 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.599921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.0.98.2:44921)
0928 09:04:55.599957 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.599959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.599988 (+ 29us) server_negotiation.cc:407] Connection header received
0928 09:04:55.600072 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.600072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.600091 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.600161 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.600307 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.600771 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.601533 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.601773 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.601857 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.602039 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.602146 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.602229 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.602268 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.602287 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:04:55.603227 8062 raft_consensus.cc:1261] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Refusing update from remote peer ae9168cd27a9458cafdde6f28beedd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.603240 8342 raft_consensus.cc:1261] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Refusing update from remote peer ae9168cd27a9458cafdde6f28beedd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.605779 8423 consensus_queue.cc:1035] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.605953 8423 consensus_queue.cc:1035] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.608126 8429 mvcc.cc:205] Tried to move back new op lower bound from 7334241466788999168 to 7334241466751692800. Current Snapshot: MvccSnapshot[applied={T|T < 7334241466788999168}]
I20260928 09:04:55.609835 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.609923 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.610553 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Sending DeleteTablet for 3 replicas of tablet 0a3578cd01d747f987c45a81116a5325
I20260928 09:04:55.611105 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a3578cd01d747f987c45a81116a5325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260928 09:04:55.611198 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a3578cd01d747f987c45a81116a5325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57556
I20260928 09:04:55.611330 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a3578cd01d747f987c45a81116a5325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42260
I20260928 09:04:55.611513 8439 tablet_replica.cc:330] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: stopping tablet replica
I20260928 09:04:55.611570 8439 raft_consensus.cc:2229] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.611611 8439 raft_consensus.cc:2258] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.611696 8440 tablet_replica.cc:330] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: stopping tablet replica
I20260928 09:04:55.611747 8440 raft_consensus.cc:2229] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.611778 8440 raft_consensus.cc:2258] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.611861 8438 tablet_replica.cc:330] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: stopping tablet replica
I20260928 09:04:55.611912 8438 raft_consensus.cc:2229] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:55.611996 8438 raft_consensus.cc:2258] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.612676 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 09:04:55.612800 8439 ts_tablet_manager.cc:1905] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 09:04:55.612826 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:04:55.614154 8440 ts_tablet_manager.cc:1905] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.615092 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7674d70f5048caa668ce3b1520fa1e (DEFAULT_TABLE table=test-table [id=27ccf2345b6d45cf8fe13aa75e52c4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.615180 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7674d70f5048caa668ce3b1520fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.615693 8440 ts_tablet_manager.cc:1918] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:55.615749 8440 log.cc:1196] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/0a3578cd01d747f987c45a81116a5325
I20260928 09:04:55.615972 8440 ts_tablet_manager.cc:1939] T 0a3578cd01d747f987c45a81116a5325 P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting consensus metadata
I20260928 09:04:55.616358 8417 tablet_bootstrap.cc:492] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Bootstrap starting.
I20260928 09:04:55.616705 8417 tablet_bootstrap.cc:654] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.617087 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7674d70f5048caa668ce3b1520fa1e (DEFAULT_TABLE table=test-table [id=27ccf2345b6d45cf8fe13aa75e52c4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.617297 7892 catalog_manager.cc:4841] TS c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627): tablet 0a3578cd01d747f987c45a81116a5325 (table test-table [id=79f1546836ea455da4b49a84775fcba0]) successfully deleted
I20260928 09:04:55.617594 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7674d70f5048caa668ce3b1520fa1e (DEFAULT_TABLE table=test-table [id=27ccf2345b6d45cf8fe13aa75e52c4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:04:55.617705 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7674d70f5048caa668ce3b1520fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.618981 8416 tablet_bootstrap.cc:492] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Bootstrap starting.
I20260928 09:04:55.619385 8416 tablet_bootstrap.cc:654] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.619995 8439 ts_tablet_manager.cc:1918] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:55.620059 8439 log.cc:1196] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/0a3578cd01d747f987c45a81116a5325
I20260928 09:04:55.620282 8439 ts_tablet_manager.cc:1939] T 0a3578cd01d747f987c45a81116a5325 P d256718298d347659b4a8bcfd60974e7: Deleting consensus metadata
I20260928 09:04:55.620412 8417 tablet_bootstrap.cc:492] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: No bootstrap required, opened a new log
I20260928 09:04:55.620419 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7674d70f5048caa668ce3b1520fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:55.620452 8417 ts_tablet_manager.cc:1397] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 09:04:55.620582 8417 raft_consensus.cc:348] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.620642 8417 raft_consensus.cc:374] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.620661 8417 raft_consensus.cc:729] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ccf5b146c4a70bbda7cf81a7d0609, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.620698 8417 consensus_queue.cc:260] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.620785 8417 ts_tablet_manager.cc:1428] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:55.621744 8418 tablet_bootstrap.cc:492] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Bootstrap starting.
I20260928 09:04:55.622030 8418 tablet_bootstrap.cc:654] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:55.622928 7892 catalog_manager.cc:4841] TS d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921): tablet 0a3578cd01d747f987c45a81116a5325 (table test-table [id=79f1546836ea455da4b49a84775fcba0]) successfully deleted
I20260928 09:04:55.623179 8438 ts_tablet_manager.cc:1905] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.624511 8438 ts_tablet_manager.cc:1918] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:04:55.624569 8438 log.cc:1196] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/0a3578cd01d747f987c45a81116a5325
I20260928 09:04:55.624782 8438 ts_tablet_manager.cc:1939] T 0a3578cd01d747f987c45a81116a5325 P ae9168cd27a9458cafdde6f28beedd6e: Deleting consensus metadata
I20260928 09:04:55.625141 8418 tablet_bootstrap.cc:492] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: No bootstrap required, opened a new log
I20260928 09:04:55.625190 8418 ts_tablet_manager.cc:1397] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 09:04:55.625207 7889 catalog_manager.cc:4841] TS ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921): tablet 0a3578cd01d747f987c45a81116a5325 (table test-table [id=79f1546836ea455da4b49a84775fcba0]) successfully deleted
I20260928 09:04:55.625344 8418 raft_consensus.cc:348] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.625445 8418 raft_consensus.cc:374] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.625474 8418 raft_consensus.cc:729] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d256718298d347659b4a8bcfd60974e7, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.625519 8418 consensus_queue.cc:260] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.625633 8418 ts_tablet_manager.cc:1428] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:55.625949 8416 tablet_bootstrap.cc:492] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: No bootstrap required, opened a new log
I20260928 09:04:55.626001 8416 ts_tablet_manager.cc:1397] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 09:04:55.626173 8416 raft_consensus.cc:348] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.626245 8416 raft_consensus.cc:374] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:55.626272 8416 raft_consensus.cc:729] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Initialized, Role: FOLLOWER
I20260928 09:04:55.626313 8416 consensus_queue.cc:260] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.626412 8416 ts_tablet_manager.cc:1428] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:55.642761 8490 raft_consensus.cc:482] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:04:55.642867 8490 raft_consensus.cc:504] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.643019 8490 leader_election.cc:288] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921), c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:55.643253 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7674d70f5048caa668ce3b1520fa1e" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" is_pre_election: true
I20260928 09:04:55.643347 8342 raft_consensus.cc:2454] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 0.
I20260928 09:04:55.643519 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7674d70f5048caa668ce3b1520fa1e" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7" is_pre_election: true
I20260928 09:04:55.643672 8062 raft_consensus.cc:2454] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 0.
I20260928 09:04:55.643934 8136 leader_election.cc:302] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, d256718298d347659b4a8bcfd60974e7; no voters:
I20260928 09:04:55.644107 8490 raft_consensus.cc:2781] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:04:55.644147 8490 raft_consensus.cc:482] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:04:55.644174 8490 raft_consensus.cc:3037] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.645159 8490 raft_consensus.cc:504] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.645279 8490 leader_election.cc:288] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 election: Requested vote from peers d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921), c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627)
I20260928 09:04:55.645663 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7674d70f5048caa668ce3b1520fa1e" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609"
I20260928 09:04:55.645743 8342 raft_consensus.cc:3037] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.646541 8342 raft_consensus.cc:2454] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 1.
I20260928 09:04:55.646754 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7674d70f5048caa668ce3b1520fa1e" candidate_uuid: "ae9168cd27a9458cafdde6f28beedd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d256718298d347659b4a8bcfd60974e7"
I20260928 09:04:55.646849 8062 raft_consensus.cc:3037] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:55.647570 8062 raft_consensus.cc:2454] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9168cd27a9458cafdde6f28beedd6e in term 1.
I20260928 09:04:55.647894 8136 leader_election.cc:302] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9168cd27a9458cafdde6f28beedd6e, d256718298d347659b4a8bcfd60974e7; no voters:
I20260928 09:04:55.648102 8490 raft_consensus.cc:2781] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:55.648151 8490 raft_consensus.cc:686] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 1 LEADER]: Becoming Leader. State: Replica: ae9168cd27a9458cafdde6f28beedd6e, State: Running, Role: LEADER
I20260928 09:04:55.648207 8490 consensus_queue.cc:237] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260928 09:04:55.648743 7904 catalog_manager.cc:5495] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ae9168cd27a9458cafdde6f28beedd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9168cd27a9458cafdde6f28beedd6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: UNKNOWN } } }
I20260928 09:04:55.665421 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.662096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37093 (local address 127.0.98.3:42627)
0928 09:04:55.662142 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.662145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.662279 (+ 134us) server_negotiation.cc:407] Connection header received
0928 09:04:55.662374 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.662375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.662385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.662414 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.662630 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.663371 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.664504 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.664811 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.664849 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.665154 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.665188 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.665215 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.665318 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.665340 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:04:55.665732 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.662064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.98.2:37093)
0928 09:04:55.662222 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.662232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.662303 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.662487 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.662487 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.662489 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:55.662560 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.662562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.663477 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.663478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.664398 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.664400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.664932 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.664933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.665023 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.665054 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.665254 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.665256 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.665601 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.665641 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260928 09:04:55.667984 8342 raft_consensus.cc:1261] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Refusing update from remote peer ae9168cd27a9458cafdde6f28beedd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.668176 8490 consensus_queue.cc:1035] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92ccf5b146c4a70bbda7cf81a7d0609" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.668433 8062 raft_consensus.cc:1261] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Refusing update from remote peer ae9168cd27a9458cafdde6f28beedd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:04:55.668807 8490 consensus_queue.cc:1035] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d256718298d347659b4a8bcfd60974e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:04:55.678798 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.675884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.98.3:41843)
0928 09:04:55.675926 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:55.675940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:55.675997 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:55.676172 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:55.676173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:55.676174 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 09:04:55.676216 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.676217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.677087 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.677088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.677962 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:55.677964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.678422 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.678423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:55.678471 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.678477 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.678643 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.678644 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 09:04:55.678679 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 09:04:55.678720 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 09:04:55.679045 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:55.676001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41843 (local address 127.0.98.2:44921)
0928 09:04:55.676042 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:55.676044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:55.676052 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:55.676075 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:55.676075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:55.676084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:55.676131 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:04:55.676284 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.676988 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.678055 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:55.678341 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:55.678394 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:55.678554 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:04:55.678586 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:04:55.678624 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:55.678960 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 09:04:55.678981 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:04:55.687247 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.687364 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:04:55.688042 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31eac824aa114125bc3151a022f950a0: Sending DeleteTablet for 3 replicas of tablet 3c7674d70f5048caa668ce3b1520fa1e
I20260928 09:04:55.688292 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7674d70f5048caa668ce3b1520fa1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260928 09:04:55.688395 8439 tablet_replica.cc:330] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: stopping tablet replica
I20260928 09:04:55.688457 8439 raft_consensus.cc:2229] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.688490 8439 raft_consensus.cc:2258] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.688699 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7674d70f5048caa668ce3b1520fa1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57556
I20260928 09:04:55.688799 8440 tablet_replica.cc:330] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: stopping tablet replica
I20260928 09:04:55.688869 8440 raft_consensus.cc:2229] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:04:55.688881 8439 ts_tablet_manager.cc:1905] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.688908 8440 raft_consensus.cc:2258] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.690634 8439 ts_tablet_manager.cc:1918] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:55.690662 8440 ts_tablet_manager.cc:1905] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.690701 8439 log.cc:1196] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/3c7674d70f5048caa668ce3b1520fa1e
I20260928 09:04:55.690696 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7674d70f5048caa668ce3b1520fa1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:04:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42260
I20260928 09:04:55.690784 8438 tablet_replica.cc:330] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: stopping tablet replica
I20260928 09:04:55.690830 8438 raft_consensus.cc:2229] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:04:55.690896 8438 raft_consensus.cc:2258] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:04:55.690958 8439 ts_tablet_manager.cc:1939] T 3c7674d70f5048caa668ce3b1520fa1e P d256718298d347659b4a8bcfd60974e7: Deleting consensus metadata
I20260928 09:04:55.691236 8438 ts_tablet_manager.cc:1905] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:04:55.691290 7892 catalog_manager.cc:4841] TS d256718298d347659b4a8bcfd60974e7 (127.0.98.1:34921): tablet 3c7674d70f5048caa668ce3b1520fa1e (table test-table [id=27ccf2345b6d45cf8fe13aa75e52c4ef]) successfully deleted
I20260928 09:04:55.691408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 7977
I20260928 09:04:55.692180 8440 ts_tablet_manager.cc:1918] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:55.692229 8440 log.cc:1196] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/3c7674d70f5048caa668ce3b1520fa1e
I20260928 09:04:55.692438 8440 ts_tablet_manager.cc:1939] T 3c7674d70f5048caa668ce3b1520fa1e P c92ccf5b146c4a70bbda7cf81a7d0609: Deleting consensus metadata
I20260928 09:04:55.692816 8438 ts_tablet_manager.cc:1918] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 09:04:55.692880 8438 log.cc:1196] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/3c7674d70f5048caa668ce3b1520fa1e
I20260928 09:04:55.693115 8438 ts_tablet_manager.cc:1939] T 3c7674d70f5048caa668ce3b1520fa1e P ae9168cd27a9458cafdde6f28beedd6e: Deleting consensus metadata
I20260928 09:04:55.693601 7892 catalog_manager.cc:4841] TS c92ccf5b146c4a70bbda7cf81a7d0609 (127.0.98.3:42627): tablet 3c7674d70f5048caa668ce3b1520fa1e (table test-table [id=27ccf2345b6d45cf8fe13aa75e52c4ef]) successfully deleted
I20260928 09:04:55.693948 7889 catalog_manager.cc:4841] TS ae9168cd27a9458cafdde6f28beedd6e (127.0.98.2:44921): tablet 3c7674d70f5048caa668ce3b1520fa1e (table test-table [id=27ccf2345b6d45cf8fe13aa75e52c4ef]) successfully deleted
I20260928 09:04:55.699875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 8117
I20260928 09:04:55.705135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 8257
I20260928 09:04:55.710850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 7873
2026-09-28T09:04:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2584 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[8509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[8509](info): set up 2 sockets
Sep 28 09:04:55 dist-test-slave-smns krb5kdc[8509](info): commencing operation
krb5kdc: starting...
W20260928 09:04:56.754256 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:56 dist-test-slave-smns krb5kdc[8509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:56.877766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40505
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:56.958667 8522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:56.958874 8522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:56.958897 8522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:56.960064 8522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:56.960107 8522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:56.960121 8522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:56.960134 8522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:56.960191 8522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:56.961697 8522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:56.961961 8522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:56.962194 8522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:56.966977 8527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:56.966979 8528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:56.967015 8522 server_base.cc:1034] running on GCE node
W20260928 09:04:56.966984 8530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:56.967381 8522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:56.967620 8522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:56.968755 8522 hybrid_clock.cc:648] HybridClock initialized: now 1790586296968744 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T09:04:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1264 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:04:57 dist-test-slave-smns krb5kdc[8535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:04:57 dist-test-slave-smns krb5kdc[8535](info): set up 2 sockets
Sep 28 09:04:57 dist-test-slave-smns krb5kdc[8535](info): commencing operation
krb5kdc: starting...
W20260928 09:04:58.022066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:04:58.141326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:58.221091 8548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:58.221344 8548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:58.221381 8548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:58.222648 8548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:58.222721 8548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:58.222745 8548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:58.222775 8548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:58.222800 8548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:58.224344 8548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:58.224628 8548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:58.224861 8548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:58.229766 8556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.229777 8554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.229812 8548 server_base.cc:1034] running on GCE node
W20260928 09:04:58.230010 8553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.230302 8548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:58.230648 8548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:58.231822 8548 hybrid_clock.cc:648] HybridClock initialized: now 1790586298231796 us; error 38 us; skew 500 ppm
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:58.234468 8548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:58.234953 8548 webserver.cc:460] Webserver started at http://127.0.98.62:44961/ using document root <none> and password file <none>
I20260928 09:04:58.235181 8548 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:58.235247 8548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:58.235383 8548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:58.236408 8548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "1d6029ad37fa4eb59aee5ea43b24092d"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "02ab255d1e468a439c157f74b6ccb89e"
server_key_iv: "4e7c97cb57f3fb6765df84bfe4f80244"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.236707 8548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d6029ad37fa4eb59aee5ea43b24092d"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "02ab255d1e468a439c157f74b6ccb89e"
server_key_iv: "4e7c97cb57f3fb6765df84bfe4f80244"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.238022 8548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:04:58.238690 8563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.238862 8548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:58.238986 8548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "1d6029ad37fa4eb59aee5ea43b24092d"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "02ab255d1e468a439c157f74b6ccb89e"
server_key_iv: "4e7c97cb57f3fb6765df84bfe4f80244"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.239065 8548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:58.255576 8548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:58.258049 8548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:58.258220 8548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:58.262068 8548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39473
I20260928 09:04:58.262122 8615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39473 every 8 connection(s)
I20260928 09:04:58.262660 8548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:58.263319 8616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:04:58.265508 8616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Bootstrap starting.
I20260928 09:04:58.265795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 8548
I20260928 09:04:58.265925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:04:58.266119 392 external_mini_cluster.cc:1428] Setting key 28810f77346ca069b63f555e9ce692b4
I20260928 09:04:58.266271 8616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Neither blocks nor log segments found. Creating new log.
I20260928 09:04:58.266628 8616 log.cc:824] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:58.267378 8616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: No bootstrap required, opened a new log
I20260928 09:04:58.269686 8616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:58.269861 8616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:04:58.269897 8616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6029ad37fa4eb59aee5ea43b24092d, State: Initialized, Role: FOLLOWER
I20260928 09:04:58.270035 8616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:58.270107 8616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:58.270128 8616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:58.270198 8616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:04:58.270764 8616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:58.270862 8616 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d6029ad37fa4eb59aee5ea43b24092d; no voters:
I20260928 09:04:58.271013 8616 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:04:58.271064 8621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:04:58.271214 8621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 LEADER]: Becoming Leader. State: Replica: 1d6029ad37fa4eb59aee5ea43b24092d, State: Running, Role: LEADER
I20260928 09:04:58.271242 8616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:58.271299 8621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:58.271584 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } } }
I20260928 09:04:58.271687 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:58.271925 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6029ad37fa4eb59aee5ea43b24092d. Latest consensus state: current_term: 1 leader_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } } }
I20260928 09:04:58.272060 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:58.272672 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:58.273320 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:58.274911 8627 catalog_manager.cc:1347] Generated new cluster ID: 98c3eeb2681a4f03a60dab3ae98c15cf
I20260928 09:04:58.274987 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:58.280442 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.266622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.98.62:39473)
0928 09:04:58.266827 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:58.266838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:58.266872 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:04:58.267428 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:58.267438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:58.267516 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:58.267657 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:58.268261 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.269739 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.270654 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.271270 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.271707 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.274290 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:58.274294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:58.274295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:58.274306 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:58.278917 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.279293 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.279296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.279296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.279389 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.279599 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.279600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.279600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.279834 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:58.280008 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:58.280112 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:04:58.280251 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260928 09:04:58.303021 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:04:58.303771 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:58.308372 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Generated new TSK 0
I20260928 09:04:58.308568 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:04:58.336673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:58.418193 8642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:58.418417 8642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:58.418440 8642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:58.419574 8642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:58.419612 8642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:58.419663 8642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:58.421206 8642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:58.421447 8642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:58.421684 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:58.426729 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.426743 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.426837 8642 server_base.cc:1034] running on GCE node
W20260928 09:04:58.426748 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.427210 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:58.427476 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:58.428627 8642 hybrid_clock.cc:648] HybridClock initialized: now 1790586298428609 us; error 33 us; skew 500 ppm
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:58.431187 8642 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:04:58.431602 8642 webserver.cc:460] Webserver started at http://127.0.98.1:40689/ using document root <none> and password file <none>
I20260928 09:04:58.431820 8642 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:58.431885 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:58.432029 8642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:58.433014 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c8ee1eb3db64271a5a119a82795d910"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "178e4b2cfeb683a601b1b9d17ac3e3fb"
server_key_iv: "15be968007b27ef5ef9235621c4f4600"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.433368 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c8ee1eb3db64271a5a119a82795d910"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "178e4b2cfeb683a601b1b9d17ac3e3fb"
server_key_iv: "15be968007b27ef5ef9235621c4f4600"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.434697 8642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:58.435482 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.435653 8642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:58.435746 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "1c8ee1eb3db64271a5a119a82795d910"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "178e4b2cfeb683a601b1b9d17ac3e3fb"
server_key_iv: "15be968007b27ef5ef9235621c4f4600"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.435828 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:58.451853 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:58.453893 8642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:58.454058 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:58.454298 8642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:58.454609 8642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:58.454660 8642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.454695 8642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:58.454722 8642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.460440 8642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40869
I20260928 09:04:58.460496 8770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40869 every 8 connection(s)
I20260928 09:04:58.460800 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:58.461496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 8642
I20260928 09:04:58.461614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:04:58.461843 392 external_mini_cluster.cc:1428] Setting key 3da46106d49ca98c2b9b93fb50e9c9d1
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 09:04:58.472620 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.461586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40717 (local address 127.0.98.62:39473)
0928 09:04:58.461638 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:58.461641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:58.462195 (+ 554us) server_negotiation.cc:407] Connection header received
0928 09:04:58.463464 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:58.463465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:58.463475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:58.463530 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:58.464366 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.465212 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.466856 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.467061 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.467118 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.469598 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:58.469600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:58.469600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:58.469609 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:58.471197 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.471627 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.471628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.471628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.471679 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.471934 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.471935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.471935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.472134 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:58.472189 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:58.472482 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 09:04:58.472554 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:04:58.472858 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.461532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:40717)
0928 09:04:58.462017 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:58.462053 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:58.463347 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:58.463601 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:58.463605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:58.464009 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:58.464275 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:58.464280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.465308 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.465309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:58.466760 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:58.466763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.467150 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.467151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:58.467309 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.467845 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:58.467856 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:58.469461 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:58.471275 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:58.471281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:58.471282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:58.471560 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:58.471776 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:58.471777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:58.471777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:58.471876 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:58.472212 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:58.472219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:58.472366 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 09:04:58.472452 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 09:04:58.472592 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":85,"threads_started":1}
I20260928 09:04:58.473471 8771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39473
I20260928 09:04:58.473598 8771 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 09:04:58.473801 8771 heartbeater.cc:511] Master 127.0.98.62:39473 requested a full tablet report, sending...
I20260928 09:04:58.474346 8580 ts_manager.cc:194] Registered new tserver with Master: 1c8ee1eb3db64271a5a119a82795d910 (127.0.98.1:40869)
I20260928 09:04:58.475101 8580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:04:58.489748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:58.570626 8776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:58.570817 8776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:58.570840 8776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:58.572021 8776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:58.572068 8776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:58.572094 8776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:58.573623 8776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:58.573863 8776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:58.574093 8776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:58.579198 8781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.579193 8782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.579193 8784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.579221 8776 server_base.cc:1034] running on GCE node
I20260928 09:04:58.579586 8776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:58.579823 8776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:58.580999 8776 hybrid_clock.cc:648] HybridClock initialized: now 1790586298580997 us; error 41 us; skew 500 ppm
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:58.583458 8776 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:04:58.583912 8776 webserver.cc:460] Webserver started at http://127.0.98.2:42619/ using document root <none> and password file <none>
I20260928 09:04:58.584151 8776 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:58.584219 8776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:58.584367 8776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:58.585382 8776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "0514fd09733d4287b8197eefd6773467"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "9bd167e7c2f9a12aea455ad1f3b63e80"
server_key_iv: "77a0b591f7f577b08a67d7d112bc3c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.585795 8776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0514fd09733d4287b8197eefd6773467"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "9bd167e7c2f9a12aea455ad1f3b63e80"
server_key_iv: "77a0b591f7f577b08a67d7d112bc3c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.587096 8776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:58.587786 8791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.587944 8776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:04:58.588038 8776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "0514fd09733d4287b8197eefd6773467"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "9bd167e7c2f9a12aea455ad1f3b63e80"
server_key_iv: "77a0b591f7f577b08a67d7d112bc3c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.588119 8776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:58.598320 8776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:58.600113 8776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:58.600284 8776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:58.600521 8776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:58.600828 8776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:58.600884 8776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.600925 8776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:58.600962 8776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.606724 8776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33781
I20260928 09:04:58.606781 8904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33781 every 8 connection(s)
I20260928 09:04:58.607100 8776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:58.614521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 8776
I20260928 09:04:58.614660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:04:58.614914 392 external_mini_cluster.cc:1428] Setting key b1fb4dcde8d38b00c06f70fbd99c14aa
I20260928 09:04:58.616922 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.607805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34305 (local address 127.0.98.62:39473)
0928 09:04:58.607863 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:58.607865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:58.608072 (+ 207us) server_negotiation.cc:407] Connection header received
0928 09:04:58.609062 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:58.609063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:58.609076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:58.609123 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:58.609817 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.610319 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.611358 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.611556 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.611630 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.613926 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:58.613927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:58.613928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:58.613935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:58.615527 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.615926 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.615927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.615928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.615986 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.616230 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.616231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.616231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.616385 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:58.616453 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:58.616699 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 09:04:58.616786 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 09:04:58.616979 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.607783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:34305)
0928 09:04:58.607949 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:58.607984 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:58.608931 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:58.609198 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:58.609201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:58.609458 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:58.609722 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:58.609727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.610412 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.610413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:58.611254 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:58.611256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.611653 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.611653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:58.611774 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.612302 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:58.612311 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:58.613836 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:58.615605 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:58.615612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:58.615615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:58.615854 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:58.616067 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:58.616068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:58.616068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:58.616164 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:58.616459 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:58.616463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:58.616575 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 09:04:58.616656 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 09:04:58.616775 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 09:04:58.617431 8905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39473
I20260928 09:04:58.617516 8905 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:58.617730 8905 heartbeater.cc:511] Master 127.0.98.62:39473 requested a full tablet report, sending...
I20260928 09:04:58.618125 8580 ts_manager.cc:194] Registered new tserver with Master: 0514fd09733d4287b8197eefd6773467 (127.0.98.2:33781)
I20260928 09:04:58.618566 8580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34305
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:04:58.641373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:58.723326 8910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:58.723523 8910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:58.723546 8910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:58.724619 8910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:58.724653 8910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:58.724678 8910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:58.726163 8910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:58.726342 8910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:58.726508 8910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:58.731513 8916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.731489 8915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.731570 8918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.731825 8910 server_base.cc:1034] running on GCE node
I20260928 09:04:58.732082 8910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:58.732331 8910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:58.733495 8910 hybrid_clock.cc:648] HybridClock initialized: now 1790586298733496 us; error 44 us; skew 500 ppm
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:58.736215 8910 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:04:58.736726 8910 webserver.cc:460] Webserver started at http://127.0.98.3:34785/ using document root <none> and password file <none>
I20260928 09:04:58.736960 8910 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:58.737028 8910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:58.737170 8910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:04:58.738314 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "80683ac0b80c48ca8f4e8eefb554004e"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "e4e2b1c57738f19765d2619202b14757"
server_key_iv: "61df5ab3a07e12459417b57f2fe6b385"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.738780 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80683ac0b80c48ca8f4e8eefb554004e"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "e4e2b1c57738f19765d2619202b14757"
server_key_iv: "61df5ab3a07e12459417b57f2fe6b385"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.740159 8910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:04:58.741118 8925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.741403 8910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:58.741483 8910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "80683ac0b80c48ca8f4e8eefb554004e"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "e4e2b1c57738f19765d2619202b14757"
server_key_iv: "61df5ab3a07e12459417b57f2fe6b385"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.741592 8910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:58.750523 8910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:58.752439 8910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:58.752597 8910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:58.752811 8910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:58.753108 8910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:58.753140 8910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.753161 8910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:58.753209 8910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.759004 8910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35921
I20260928 09:04:58.759102 9038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35921 every 8 connection(s)
I20260928 09:04:58.759438 8910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:04:58.766681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 8910
I20260928 09:04:58.766836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:04:58.767060 392 external_mini_cluster.cc:1428] Setting key cec89bef5d12dbbd4ff84bb8289b6d7d
I20260928 09:04:58.769284 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.760170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51849 (local address 127.0.98.62:39473)
0928 09:04:58.760213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:58.760215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:58.760483 (+ 268us) server_negotiation.cc:407] Connection header received
0928 09:04:58.761454 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:58.761456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:58.761468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:58.761523 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:58.762176 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.762689 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.763731 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.763945 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.764035 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.766155 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:58.766156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:58.766156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:58.766171 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:58.767848 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.768313 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.768314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.768314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.768371 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:58.768633 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:58.768633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:58.768634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:58.768748 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:58.768846 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:58.769113 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 09:04:58.769189 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:04:58.769472 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:58.760141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:51849)
0928 09:04:58.760315 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:58.760393 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:58.761336 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:58.761633 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:58.761636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:58.761885 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:58.762077 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:58.762081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.762788 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.762790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:58.763629 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:58.763632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:58.764045 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:58.764047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:58.764167 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:58.764697 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:58.764706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:58.766044 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:04:58.767945 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:58.767952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:58.767954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:58.768231 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:58.768441 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:04:58.768442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:04:58.768442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:04:58.768556 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:04:58.768837 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:04:58.768843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:04:58.768982 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 09:04:58.769094 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 09:04:58.769247 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 09:04:58.769888 9039 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39473
I20260928 09:04:58.769999 9039 heartbeater.cc:464] Registering TS with master...
I20260928 09:04:58.770205 9039 heartbeater.cc:511] Master 127.0.98.62:39473 requested a full tablet report, sending...
I20260928 09:04:58.770537 8580 ts_manager.cc:194] Registered new tserver with Master: 80683ac0b80c48ca8f4e8eefb554004e (127.0.98.3:35921)
I20260928 09:04:58.770968 8580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51849
I20260928 09:04:58.778925 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 09:04:58.856369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 8642
I20260928 09:04:58.861073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 8776
I20260928 09:04:58.865442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 8910
I20260928 09:04:58.869601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 8548
I20260928 09:04:58.874593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39473
--webserver_interface=127.0.98.62
--webserver_port=44961
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:04:58.955567 9048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:58.955781 9048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:58.955803 9048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:58.957248 9048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:04:58.957305 9048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:58.957320 9048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:58.957333 9048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:04:58.957345 9048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:04:58.959395 9048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:58.959602 9048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:58.959772 9048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:58.964740 9056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.964784 9054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:58.964898 9053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:58.965281 9048 server_base.cc:1034] running on GCE node
I20260928 09:04:58.965520 9048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:58.965816 9048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:58.966953 9048 hybrid_clock.cc:648] HybridClock initialized: now 1790586298966928 us; error 49 us; skew 500 ppm
Sep 28 09:04:58 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:58.969435 9048 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:04:58.969903 9048 webserver.cc:460] Webserver started at http://127.0.98.62:44961/ using document root <none> and password file <none>
I20260928 09:04:58.970139 9048 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:58.970222 9048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:58.971607 9048 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:58.972350 9063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:58.972571 9048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:04:58.972661 9048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "1d6029ad37fa4eb59aee5ea43b24092d"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "02ab255d1e468a439c157f74b6ccb89e"
server_key_iv: "4e7c97cb57f3fb6765df84bfe4f80244"
server_key_version: "encryptionkey@0"
I20260928 09:04:58.972991 9048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:58.981745 9048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:58.983731 9048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:58.983852 9048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:58.987725 9048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39473
I20260928 09:04:58.987752 9115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39473 every 8 connection(s)
I20260928 09:04:58.988313 9048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:04:58.988952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9048
I20260928 09:04:58.989257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40869
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39473
--webserver_port=40689
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:04:58.989765 9116 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:04:58.990396 9116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Bootstrap starting.
I20260928 09:04:58.998903 9116 log.cc:824] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:04:59.000676 9116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:04:59.000982 9116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Bootstrap complete.
I20260928 09:04:59.003276 9116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:59.003539 9116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6029ad37fa4eb59aee5ea43b24092d, State: Initialized, Role: FOLLOWER
I20260928 09:04:59.003752 9116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:59.003851 9116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:04:59.003906 9116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:04:59.003953 9116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:04:59.004993 9116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:59.005122 9116 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d6029ad37fa4eb59aee5ea43b24092d; no voters:
I20260928 09:04:59.005342 9116 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:04:59.005384 9120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:04:59.005504 9120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [term 2 LEADER]: Becoming Leader. State: Replica: 1d6029ad37fa4eb59aee5ea43b24092d, State: Running, Role: LEADER
I20260928 09:04:59.005616 9120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260928 09:04:59.005647 9116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:04:59.005913 9121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } } }
I20260928 09:04:59.006076 9121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:59.006282 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6029ad37fa4eb59aee5ea43b24092d. Latest consensus state: current_term: 2 leader_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6029ad37fa4eb59aee5ea43b24092d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } } }
I20260928 09:04:59.006385 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d [sys.catalog]: This master's current role is: LEADER
I20260928 09:04:59.006461 9127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:04:59.007037 9127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:04:59.007193 9127 catalog_manager.cc:1259] Loaded cluster ID: 98c3eeb2681a4f03a60dab3ae98c15cf
I20260928 09:04:59.007236 9127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:04:59.008108 9127 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:04:59.008301 9127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d6029ad37fa4eb59aee5ea43b24092d: Loaded TSK: 0
I20260928 09:04:59.008518 9127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:04:59.083819 9118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:59.084025 9118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:59.084049 9118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:59.085263 9118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:59.085305 9118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:59.085340 9118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:04:59.086844 9118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40689
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:59.087073 9118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:59.087324 9118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:59.092363 9142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:59.092496 9118 server_base.cc:1034] running on GCE node
W20260928 09:04:59.092403 9143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:59.092667 9145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:59.092902 9118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:59.093154 9118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:59.094338 9118 hybrid_clock.cc:648] HybridClock initialized: now 1790586299094315 us; error 41 us; skew 500 ppm
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586299, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:59.097172 9118 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 09:04:59.097746 9118 webserver.cc:460] Webserver started at http://127.0.98.1:40689/ using document root <none> and password file <none>
I20260928 09:04:59.097981 9118 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:59.098064 9118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:59.099545 9118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:59.100203 9152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.100376 9118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:59.100499 9118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "1c8ee1eb3db64271a5a119a82795d910"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "178e4b2cfeb683a601b1b9d17ac3e3fb"
server_key_iv: "15be968007b27ef5ef9235621c4f4600"
server_key_version: "encryptionkey@0"
I20260928 09:04:59.100895 9118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:59.119040 9118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:59.121320 9118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:59.121515 9118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:59.121801 9118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:59.122174 9118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:59.122225 9118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.122269 9118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:59.122295 9118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.128609 9118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40869
I20260928 09:04:59.128657 9265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40869 every 8 connection(s)
I20260928 09:04:59.129000 9118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:04:59.134608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9118
I20260928 09:04:59.134941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33781
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39473
--webserver_port=42619
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.138127 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.129793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:50385)
0928 09:04:59.129992 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.130032 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.130602 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.131041 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.131045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.131641 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.131828 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.131833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.132972 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.132973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.133872 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.133874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.134660 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.134661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.134814 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.135427 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.135438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.137906 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260928 09:04:59.138372 9266 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39473: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260928 09:04:59.138494 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.130062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50385 (local address 127.0.98.62:39473)
0928 09:04:59.130259 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.130269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.130310 (+ 41us) server_negotiation.cc:407] Connection header received
0928 09:04:59.130808 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.130813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.130865 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.131001 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.131946 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.132855 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.133991 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.134444 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.134562 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.138028 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50385: BlockingRecv error: recv got EOF from 127.0.98.1:50385 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.142908 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.138587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:49283)
0928 09:04:59.138768 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.138778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.138849 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.138978 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.138979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.139106 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.139150 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.139152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.139794 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.139795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.140311 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.140313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.140684 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.140685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.140755 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.141185 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.141190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.142824 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
W20260928 09:04:59.142966 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.138789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49283 (local address 127.0.98.62:39473)
0928 09:04:59.138850 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.138852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.138861 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:59.138908 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.138909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.138922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.138976 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.139229 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.139707 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.140402 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.140596 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.140682 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.142892 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49283: BlockingRecv error: recv got EOF from 127.0.98.1:49283 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.147524 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.143397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:51249)
0928 09:04:59.143436 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.143445 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.143480 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.143640 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.143640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.143764 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.143790 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.143791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.144412 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.144413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.144910 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.144911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.145285 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.145286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.145338 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.145829 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.145834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.147435 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 09:04:59.147560 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.143440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51249 (local address 127.0.98.62:39473)
0928 09:04:59.143543 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.143545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.143550 (+ 5us) server_negotiation.cc:407] Connection header received
0928 09:04:59.143567 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.143567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.143574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.143637 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.143867 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.144332 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.145016 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.145200 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.145283 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.147505 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51249: BlockingRecv error: recv got EOF from 127.0.98.1:51249 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 09:04:59.147877 9266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:04:59.223898 9270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:59.224104 9270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:59.224128 9270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:59.225224 9270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:59.225255 9270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:59.225281 9270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:04:59.226626 9270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42619
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:59.226822 9270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:59.226991 9270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:04:59.231830 9270 server_base.cc:1034] running on GCE node
W20260928 09:04:59.231798 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:59.231889 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:59.231798 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:59.232132 9270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:59.232316 9270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:59.233453 9270 hybrid_clock.cc:648] HybridClock initialized: now 1790586299233437 us; error 30 us; skew 500 ppm
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586299, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:59.235977 9270 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 09:04:59.236416 9270 webserver.cc:460] Webserver started at http://127.0.98.2:42619/ using document root <none> and password file <none>
I20260928 09:04:59.236639 9270 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:59.236724 9270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:59.238164 9270 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:59.238826 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.239013 9270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:59.239120 9270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "0514fd09733d4287b8197eefd6773467"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "9bd167e7c2f9a12aea455ad1f3b63e80"
server_key_iv: "77a0b591f7f577b08a67d7d112bc3c30"
server_key_version: "encryptionkey@0"
I20260928 09:04:59.239488 9270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:59.247043 9270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:59.248973 9270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:59.249137 9270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:59.249357 9270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:59.249773 9270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:59.249828 9270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.249871 9270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:59.249903 9270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.256111 9270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33781
I20260928 09:04:59.256152 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33781 every 8 connection(s)
I20260928 09:04:59.256494 9270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:04:59.260619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9270
I20260928 09:04:59.260924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35921
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39473
--webserver_port=34785
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 09:04:59.264554 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.257182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38195 (local address 127.0.98.62:39473)
0928 09:04:59.257285 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.257288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.257485 (+ 197us) server_negotiation.cc:407] Connection header received
0928 09:04:59.258093 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.258094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.258111 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.258179 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.259162 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.259666 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.260708 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.260913 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.260991 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.264474 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38195: BlockingRecv error: recv got EOF from 127.0.98.2:38195 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:04:59.264591 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.257130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:38195)
0928 09:04:59.257357 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.257393 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.257970 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.258253 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.258257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.258861 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.259050 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.259055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.259761 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.259762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.260603 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.260606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.260996 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.260997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.261127 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.261696 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.261704 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.264375 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
W20260928 09:04:59.264816 9400 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39473: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.269459 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.265036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:33851)
0928 09:04:59.265089 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.265100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.265147 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.265282 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.265283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.265413 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.265456 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.265457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.266122 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.266123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.266636 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.266637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.267011 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.267012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.267076 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.267555 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.267559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.269373 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 09:04:59.269531 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.265104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33851 (local address 127.0.98.62:39473)
0928 09:04:59.265163 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.265164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.265173 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:04:59.265200 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.265201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.265210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.265280 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.265577 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.266040 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.266720 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.266922 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.267013 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.269446 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33851: BlockingRecv error: recv got EOF from 127.0.98.2:33851 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.274343 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.269728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:49139)
0928 09:04:59.269870 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.269880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.269933 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.270072 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.270072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.270237 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.270277 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.270278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.270892 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.270893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.271446 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.271448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.271844 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.271845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.271910 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.272381 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.272385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.274249 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260928 09:04:59.274375 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.269881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49139 (local address 127.0.98.62:39473)
0928 09:04:59.269944 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.269946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.269956 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:04:59.269979 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.269980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.269988 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.270069 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.270362 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.270810 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.271556 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.271742 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.271825 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.274331 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49139: BlockingRecv error: recv got EOF from 127.0.98.2:49139 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 09:04:59.274501 9400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:04:59.346009 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:04:59.346226 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:04:59.346249 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:04:59.347399 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:04:59.347443 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:04:59.347469 9403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:04:59.349123 9403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34785
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:04:59.349334 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:04:59.349524 9403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:04:59.354336 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:59.354331 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:04:59.354336 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:04:59.354640 9403 server_base.cc:1034] running on GCE node
I20260928 09:04:59.354843 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:04:59.355108 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:04:59.356247 9403 hybrid_clock.cc:648] HybridClock initialized: now 1790586299356225 us; error 37 us; skew 500 ppm
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586299, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:04:59.358850 9403 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 09:04:59.359339 9403 webserver.cc:460] Webserver started at http://127.0.98.3:34785/ using document root <none> and password file <none>
I20260928 09:04:59.359583 9403 fs_manager.cc:359] Metadata directory not provided
I20260928 09:04:59.359666 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:04:59.360977 9403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:04:59.361605 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.361774 9403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:04:59.361871 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "80683ac0b80c48ca8f4e8eefb554004e"
format_stamp: "Formatted at 2026-09-28 09:04:58 on dist-test-slave-smns"
server_key: "e4e2b1c57738f19765d2619202b14757"
server_key_iv: "61df5ab3a07e12459417b57f2fe6b385"
server_key_version: "encryptionkey@0"
I20260928 09:04:59.362212 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:04:59.390311 9403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:04:59.392735 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:04:59.392910 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:04:59.393165 9403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:04:59.393471 9403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:04:59.393529 9403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.393584 9403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:04:59.393635 9403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:04:59.399641 9403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35921
I20260928 09:04:59.399703 9532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35921 every 8 connection(s)
I20260928 09:04:59.400032 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.406705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9403
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 09:04:59.407787 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.400735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52671 (local address 127.0.98.62:39473)
0928 09:04:59.400815 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.400817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.401004 (+ 187us) server_negotiation.cc:407] Connection header received
0928 09:04:59.401485 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.401486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.401496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.401583 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.402548 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.403081 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.404192 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.404370 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.404426 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.407713 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52671: BlockingRecv error: recv got EOF from 127.0.98.3:52671 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:04:59.407860 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.400701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:52671)
0928 09:04:59.400880 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.400918 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.401397 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.401673 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.401676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.402245 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.402459 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.402463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.403156 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.403157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.404123 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.404126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.404439 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.404440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.404566 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.405040 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.405048 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.407626 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
W20260928 09:04:59.408094 9533 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39473: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.412762 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.408326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:56269)
0928 09:04:59.408387 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.408398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.408439 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.408567 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.408568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.408696 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.408735 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.408736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.409350 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.409351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.409929 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.409931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.410299 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.410300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.410365 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.410868 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.410873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.412717 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 09:04:59.412889 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.408378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56269 (local address 127.0.98.62:39473)
0928 09:04:59.408436 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.408438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.408446 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:04:59.408492 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.408492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.408501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.408562 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.408809 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.409266 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.410010 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.410215 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.410298 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.412813 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56269: BlockingRecv error: recv got EOF from 127.0.98.3:56269 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:04:59.415405 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.407303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.98.62:39473)
0928 09:04:59.407435 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.407437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.407561 (+ 124us) server_negotiation.cc:407] Connection header received
0928 09:04:59.407650 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.407651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.407663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.407730 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.408562 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.409052 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.409905 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.410119 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.410204 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.411719 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:04:59.411724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:04:59.411724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:04:59.411736 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:04:59.413910 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:59.414304 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:59.414307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:59.414307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:59.414425 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:04:59.414625 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:04:59.414626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:04:59.414626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:04:59.414918 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:04:59.415056 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 09:04:59.415183 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 09:04:59.415301 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:04:59 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:04:59.417662 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:04:59.413292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:38001)
0928 09:04:59.413335 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 09:04:59.413345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:04:59.413379 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:04:59.413504 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:04:59.413505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:04:59.413651 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:04:59.413683 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.413685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.414412 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.414413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.415161 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:04:59.415163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.415590 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.415591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:04:59.415638 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.416039 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:04:59.416044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:04:59.417622 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:04:59.417754 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:04:59.413363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38001 (local address 127.0.98.62:39473)
0928 09:04:59.413421 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:04:59.413423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:04:59.413430 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:04:59.413446 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:04:59.413446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:04:59.413453 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:04:59.413499 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:04:59.413756 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.414344 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.415245 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:04:59.415442 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:04:59.415536 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:04:59.417695 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38001: BlockingRecv error: recv got EOF from 127.0.98.3:38001 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 09:04:59.417806 9533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 09:05:00 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:00 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:00.156801 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:00.148246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:43043)
0928 09:05:00.148396 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:00.148410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:00.148501 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:00.148641 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:00.148642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:00.148804 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:00.148870 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:00.148872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.149529 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.149529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:00.150129 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:00.150131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.150475 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.150476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:00.150523 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:00.154899 (+ 4376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:00.154905 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:00.156735 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260928 09:05:00.156919 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:00.148324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43043 (local address 127.0.98.62:39473)
0928 09:05:00.148466 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:00.148468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:00.148486 (+ 18us) server_negotiation.cc:407] Connection header received
0928 09:05:00.148560 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:00.148560 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:00.148570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:00.148629 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:00.148956 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.149453 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.150204 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.150404 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.150461 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:00.156837 (+ 6376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43043: BlockingRecv error: recv got EOF from 127.0.98.1:43043 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
Sep 28 09:05:00 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:00 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:00.279489 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:00.274827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:58675)
0928 09:05:00.274985 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:00.274998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:00.275064 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:00.275193 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:00.275194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:00.275370 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:00.275416 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:00.275418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.276179 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.276180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:00.276747 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:00.276749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.277144 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.277145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:00.277223 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:00.277717 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:00.277722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:00.279366 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
W20260928 09:05:00.279500 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:00.274896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58675 (local address 127.0.98.62:39473)
0928 09:05:00.274991 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:00.274994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:00.275044 (+ 50us) server_negotiation.cc:407] Connection header received
0928 09:05:00.275122 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:00.275123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:00.275136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:00.275190 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:00.275509 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.276077 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.276836 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.277041 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.277147 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:00.279448 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58675: BlockingRecv error: recv got EOF from 127.0.98.2:58675 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 09:05:00 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:00 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:00.422822 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:00.418188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:33793)
0928 09:05:00.418315 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:00.418329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:00.418407 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:00.418538 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:00.418539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:00.418702 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:00.418750 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:00.418752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.419389 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.419390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:00.419971 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:00.419978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.420307 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.420307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:00.420346 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:00.420819 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:00.420825 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:00.422753 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
W20260928 09:05:00.422956 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:00.418284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33793 (local address 127.0.98.62:39473)
0928 09:05:00.418343 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:00.418345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:00.418391 (+ 46us) server_negotiation.cc:407] Connection header received
0928 09:05:00.418452 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:00.418453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:00.418470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:00.418517 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:00.418818 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.419313 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.420060 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:00.420241 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:00.420293 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:00.422853 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33793: BlockingRecv error: recv got EOF from 127.0.98.3:33793 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 09:05:01 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:01 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:01.162597 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:01.157318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:41289)
0928 09:05:01.157456 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:01.157469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:01.157535 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:01.157737 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:01.157738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:01.157954 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:01.158021 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:01.158024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.158691 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.158692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:01.159661 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:01.159664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.160013 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.160014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:01.160080 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:01.160569 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:01.160575 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:01.162539 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260928 09:05:01.162755 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:01.157405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41289 (local address 127.0.98.62:39473)
0928 09:05:01.157507 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:01.157509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:01.157520 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:05:01.157630 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:01.157631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:01.157643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:01.157709 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:01.158098 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.158608 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.159748 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.159937 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.159987 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:01.162663 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41289: BlockingRecv error: recv got EOF from 127.0.98.1:41289 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
Sep 28 09:05:01 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:01 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:01.284372 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:01.279944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:48565)
0928 09:05:01.280099 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:01.280116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:01.280171 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:01.280302 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:01.280303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:01.280459 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:01.280526 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:01.280528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.281199 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.281200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:01.281745 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:01.281747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.282082 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.282083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:01.282121 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:01.282556 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:01.282561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:01.284312 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
W20260928 09:05:01.284538 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:01.280039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48565 (local address 127.0.98.62:39473)
0928 09:05:01.280108 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:01.280110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:01.280158 (+ 48us) server_negotiation.cc:407] Connection header received
0928 09:05:01.280227 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:01.280228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:01.280238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:01.280280 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:01.280595 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.281131 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.281810 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.282020 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.282073 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:01.284453 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48565: BlockingRecv error: recv got EOF from 127.0.98.2:48565 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 09:05:01 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:01 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:01.428215 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:01.423366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:60369)
0928 09:05:01.423533 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:01.423546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:01.423645 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:01.423822 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:01.423823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:01.424035 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:01.424084 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:01.424086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.424786 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.424787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:01.425317 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:01.425318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.425840 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.425841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:01.425929 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:01.426405 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:01.426410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:01.428157 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
W20260928 09:05:01.428362 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:01.423548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60369 (local address 127.0.98.62:39473)
0928 09:05:01.423613 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:01.423616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:01.423625 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:05:01.423717 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:01.423718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:01.423740 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:01.423802 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:01.424170 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.424695 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.425421 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:01.425733 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:01.425803 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:01.428282 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60369: BlockingRecv error: recv got EOF from 127.0.98.3:60369 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 09:05:02 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:02 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:02.168187 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:02.163247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:56761)
0928 09:05:02.163485 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:02.163496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:02.163575 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:02.163759 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:02.163761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:02.163920 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:02.163974 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:02.163976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.164755 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.164755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:02.165311 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:02.165313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.165779 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.165780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:02.165826 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:02.166260 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:02.166265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:02.168128 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":150,"threads_started":1}
W20260928 09:05:02.168352 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:02.163303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56761 (local address 127.0.98.62:39473)
0928 09:05:02.163449 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:02.163452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:02.163570 (+ 118us) server_negotiation.cc:407] Connection header received
0928 09:05:02.163647 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:02.163647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:02.163659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:02.163718 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:02.164067 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.164685 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.165373 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.165696 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.165754 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:02.168240 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56761: BlockingRecv error: recv got EOF from 127.0.98.1:56761 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
Sep 28 09:05:02 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:02 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:02.289690 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:02.284800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:49975)
0928 09:05:02.284931 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:02.284943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:02.284994 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:02.285138 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:02.285138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:02.285292 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:02.285344 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:02.285345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.286196 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.286198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:02.286735 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:02.286737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.287162 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.287163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:02.287204 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:02.287654 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:02.287659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:02.289630 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
W20260928 09:05:02.289810 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:02.284896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49975 (local address 127.0.98.62:39473)
0928 09:05:02.284971 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:02.284973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:02.285001 (+ 28us) server_negotiation.cc:407] Connection header received
0928 09:05:02.285068 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:02.285068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:02.285079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:02.285123 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:02.285444 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.286084 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.286895 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.287088 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.287147 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:02.289738 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49975: BlockingRecv error: recv got EOF from 127.0.98.2:49975 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 09:05:02 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:02 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:02.433452 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:02.428662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:58867)
0928 09:05:02.428852 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:02.428865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:02.428936 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:02.429113 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:02.429114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:02.429302 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:02.429350 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:02.429351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.430102 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.430103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:02.430661 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:02.430664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.431061 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.431062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:02.431125 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:02.431628 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:02.431634 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:02.433381 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
W20260928 09:05:02.433606 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:02.428761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58867 (local address 127.0.98.62:39473)
0928 09:05:02.428836 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:02.428838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:02.428916 (+ 78us) server_negotiation.cc:407] Connection header received
0928 09:05:02.429026 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:02.429026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:02.429039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:02.429107 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:02.429434 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.430011 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.430754 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:02.430948 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:02.431046 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:02.433476 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58867: BlockingRecv error: recv got EOF from 127.0.98.3:58867 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 09:05:03 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:03 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:03.173329 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:03.168669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:48213)
0928 09:05:03.168812 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:03.168831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:03.168899 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:03.169059 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:03.169060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:03.169220 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:03.169270 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:03.169271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.170044 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.170046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:03.170616 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:03.170618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.170973 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.170974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:03.171019 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:03.171480 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:03.171485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:03.173270 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
W20260928 09:05:03.173475 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:03.168711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48213 (local address 127.0.98.62:39473)
0928 09:05:03.168856 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:03.168858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:03.168881 (+ 23us) server_negotiation.cc:407] Connection header received
0928 09:05:03.168942 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:03.168943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:03.168983 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:03.169044 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:03.169332 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.169925 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.170693 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.170884 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.170934 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:03.173382 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48213: BlockingRecv error: recv got EOF from 127.0.98.1:48213 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
Sep 28 09:05:03 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:03 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:03.294523 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:03.290193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:40923)
0928 09:05:03.290307 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:03.290318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:03.290371 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:03.290490 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:03.290490 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:03.290647 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:03.290693 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:03.290694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.291322 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.291323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:03.291906 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:03.291908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.292245 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.292246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:03.292284 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:03.292706 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:03.292711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:03.294445 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
W20260928 09:05:03.294608 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:03.290262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40923 (local address 127.0.98.62:39473)
0928 09:05:03.290341 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:03.290343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:03.290355 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:05:03.290428 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:03.290428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:03.290439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:03.290487 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:03.290747 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.291247 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.291982 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.292182 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.292237 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:03.294544 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40923: BlockingRecv error: recv got EOF from 127.0.98.2:40923 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 09:05:03 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:03 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:03.438750 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:03.434115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:38359)
0928 09:05:03.434261 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:03.434271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:03.434327 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:03.434461 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:03.434462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:03.434617 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:03.434665 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:03.434667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.435327 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.435328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:03.435866 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:03.435868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.436261 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.436262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:03.436325 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:03.436809 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:03.436814 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:03.438656 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
W20260928 09:05:03.438791 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:03.434206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38359 (local address 127.0.98.62:39473)
0928 09:05:03.434255 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:03.434257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:03.434312 (+ 55us) server_negotiation.cc:407] Connection header received
0928 09:05:03.434389 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:03.434390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:03.434401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:03.434445 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:03.434727 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.435212 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.435960 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:03.436174 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:03.436247 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:03.438729 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38359: BlockingRecv error: recv got EOF from 127.0.98.3:38359 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 09:05:04 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:04 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:04.178849 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:04.173832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:53277)
0928 09:05:04.173987 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:04.173998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:04.174064 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:04.174203 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:04.174203 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:04.174361 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:04.174445 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:04.174447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.175341 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.175342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:04.175944 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:04.175946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.176360 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.176361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:04.176405 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:04.176888 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:04.176893 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:04.178788 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
W20260928 09:05:04.179018 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:04.173902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53277 (local address 127.0.98.62:39473)
0928 09:05:04.174012 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:04.174013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:04.174049 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:05:04.174117 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:04.174117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:04.174127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:04.174186 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:04.174526 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.175265 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.176051 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.176300 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.176354 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:04.178884 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53277: BlockingRecv error: recv got EOF from 127.0.98.1:53277 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
Sep 28 09:05:04 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:04 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:04.299296 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:04.294974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:51655)
0928 09:05:04.295098 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:04.295110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:04.295165 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:04.295305 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:04.295306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:04.295460 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:04.295508 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:04.295509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.296128 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.296129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:04.296673 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:04.296675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.297015 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.297016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:04.297055 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:04.297485 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:04.297490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:04.299227 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260928 09:05:04.299435 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:04.295121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51655 (local address 127.0.98.62:39473)
0928 09:05:04.295178 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:04.295180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:04.295213 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:05:04.295237 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:04.295237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:04.295247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:04.295289 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:04.295572 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.296065 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.296745 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.296926 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.296978 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:04.299343 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51655: BlockingRecv error: recv got EOF from 127.0.98.2:51655 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 09:05:04 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:04 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:04.443894 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:04.439319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:51393)
0928 09:05:04.439465 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:04.439480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:04.439560 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:04.439716 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:04.439717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:04.439881 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:04.439927 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:04.439929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.440609 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.440610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:04.441160 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:04.441162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.441521 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.441522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:04.441606 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:04.442126 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:04.442131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:04.443837 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
W20260928 09:05:04.443980 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:04.439378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51393 (local address 127.0.98.62:39473)
0928 09:05:04.439450 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:04.439453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:04.439540 (+ 87us) server_negotiation.cc:407] Connection header received
0928 09:05:04.439627 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:04.439628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:04.439642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:04.439711 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:04.440003 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.440520 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.441242 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:04.441436 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:04.441522 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:04.443925 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51393: BlockingRecv error: recv got EOF from 127.0.98.3:51393 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 09:05:05 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:05 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:05.190781 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:05.179456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:54471)
0928 09:05:05.179569 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:05.179584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:05.179636 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:05.179799 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:05.179800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:05.180012 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:05.180063 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:05.180065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.180701 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.180701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:05.181234 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:05.181235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.181696 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.181697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:05.181738 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:05.188701 (+ 6963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:05.188707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:05.190713 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260928 09:05:05.190881 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:05.179491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54471 (local address 127.0.98.62:39473)
0928 09:05:05.179641 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:05.179643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:05.179655 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:05:05.179710 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:05.179711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:05.179722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:05.179792 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:05.180133 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.180633 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.181318 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.181523 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.181616 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:05.190787 (+ 9171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54471: BlockingRecv error: recv got EOF from 127.0.98.1:54471 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
Sep 28 09:05:05 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:05 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:05.304553 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:05.299803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:46063)
0928 09:05:05.299983 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:05.299994 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:05.300074 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:05.300422 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:05.300422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:05.300589 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:05.300638 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:05.300640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.301346 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.301347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:05.301916 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:05.301917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.302271 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.302271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:05.302310 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:05.302738 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:05.302742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:05.304491 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
W20260928 09:05:05.304684 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:05.299892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46063 (local address 127.0.98.62:39473)
0928 09:05:05.300023 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:05.300024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:05.300073 (+ 49us) server_negotiation.cc:407] Connection header received
0928 09:05:05.300160 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:05.300161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:05.300172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:05.300405 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:05.300717 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.301274 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.302008 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.302208 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.302263 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:05.304600 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46063: BlockingRecv error: recv got EOF from 127.0.98.2:46063 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 09:05:05 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:05 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:05.448886 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:05.444414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:43467)
0928 09:05:05.444536 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:05.444552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:05.444609 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:05.444759 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:05.444761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:05.444917 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:05.444969 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:05.444970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.445654 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.445654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:05.446176 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:05.446177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.446526 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.446527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:05.446576 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:05.447025 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:05.447030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:05.448816 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260928 09:05:05.448993 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:05.444481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43467 (local address 127.0.98.62:39473)
0928 09:05:05.444559 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:05.444561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:05.444604 (+ 43us) server_negotiation.cc:407] Connection header received
0928 09:05:05.444652 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:05.444653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:05.444664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:05.444722 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:05.445055 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.445587 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.446243 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:05.446423 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:05.446474 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:05.448903 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43467: BlockingRecv error: recv got EOF from 127.0.98.3:43467 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 09:05:06 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:06 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:06.196236 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:06.191294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:37387)
0928 09:05:06.191425 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:06.191437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:06.191496 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:06.191667 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:06.191668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:06.191832 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:06.191884 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:06.191886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.192578 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.192579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:06.193114 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:06.193115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.193509 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.193510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:06.193571 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:06.194016 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:06.194021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:06.196122 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
W20260928 09:05:06.196291 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:06.191380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37387 (local address 127.0.98.62:39473)
0928 09:05:06.191520 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:06.191522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:06.191537 (+ 15us) server_negotiation.cc:407] Connection header received
0928 09:05:06.191562 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:06.191562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:06.191573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:06.191653 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:06.191970 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.192501 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.193194 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.193414 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.193495 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:06.196207 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37387: BlockingRecv error: recv got EOF from 127.0.98.1:37387 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Sep 28 09:05:06 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:06 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:06.309798 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:06.305040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:40745)
0928 09:05:06.305200 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:06.305213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:06.305281 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:06.305411 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:06.305412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:06.305702 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:06.305752 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:06.305754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.306528 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.306529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:06.307140 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:06.307142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.307523 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.307524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:06.307582 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:06.308024 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:06.308029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:06.309701 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
W20260928 09:05:06.309852 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:06.305137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40745 (local address 127.0.98.62:39473)
0928 09:05:06.305205 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:06.305208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:06.305261 (+ 53us) server_negotiation.cc:407] Connection header received
0928 09:05:06.305343 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:06.305344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:06.305357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:06.305409 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:06.305839 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.306448 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.307223 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.307436 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.307519 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:06.309788 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40745: BlockingRecv error: recv got EOF from 127.0.98.2:40745 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 09:05:06 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:06 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:06.454212 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:06.449366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:37683)
0928 09:05:06.449475 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:06.449487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:06.449583 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:06.449717 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:06.449717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:06.449894 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:06.449940 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:06.449943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.450659 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.450661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:06.451216 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:06.451218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.451587 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.451588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:06.451645 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:06.452162 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:06.452166 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:06.454115 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
W20260928 09:05:06.454254 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:06.449434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37683 (local address 127.0.98.62:39473)
0928 09:05:06.449511 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:06.449513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:06.449540 (+ 27us) server_negotiation.cc:407] Connection header received
0928 09:05:06.449639 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:06.449639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:06.449653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:06.449710 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:06.450058 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.450572 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.451310 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:06.451519 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:06.451590 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:06.454192 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37683: BlockingRecv error: recv got EOF from 127.0.98.3:37683 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 09:05:07 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:07 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:07.201432 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:07.196711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:41189)
0928 09:05:07.196962 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:07.196973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:07.197025 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:07.197260 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:07.197261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:07.197421 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:07.197469 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:07.197470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.198232 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.198233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:07.198779 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:07.198781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.199170 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.199171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:07.199209 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:07.199633 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:07.199637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:07.201374 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":199,"threads_started":1}
W20260928 09:05:07.201592 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:07.196940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41189 (local address 127.0.98.62:39473)
0928 09:05:07.197139 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:07.197142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:07.197151 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:05:07.197173 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:07.197173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:07.197183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:07.197249 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:07.197567 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.198152 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.198893 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.199104 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.199165 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:07.201482 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41189: BlockingRecv error: recv got EOF from 127.0.98.1:41189 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":145,"threads_started":1}
Sep 28 09:05:07 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:07 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:07.314906 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:07.310262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:56325)
0928 09:05:07.310434 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:07.310444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:07.310502 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:07.310633 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:07.310633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:07.310790 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:07.310839 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:07.310840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.311602 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.311603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:07.312135 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:07.312137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.312488 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.312489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:07.312540 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:07.313116 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:07.313120 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:07.314808 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
W20260928 09:05:07.314957 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:07.310337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56325 (local address 127.0.98.62:39473)
0928 09:05:07.310402 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:07.310404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:07.310497 (+ 93us) server_negotiation.cc:407] Connection header received
0928 09:05:07.310551 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:07.310552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:07.310564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:07.310620 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:07.310892 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.311527 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.312203 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.312399 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.312449 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:07.314891 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56325: BlockingRecv error: recv got EOF from 127.0.98.2:56325 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 09:05:07 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:07 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:07.459264 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:07.454720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:52359)
0928 09:05:07.454899 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:07.454909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:07.454967 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:07.455096 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:07.455097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:07.455259 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:07.455327 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:07.455329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.455964 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.455965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:07.456515 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:07.456516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.456826 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.456827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:07.456866 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:07.457289 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:07.457294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:07.459166 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":102,"threads_started":1}
W20260928 09:05:07.459318 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:07.454806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52359 (local address 127.0.98.62:39473)
0928 09:05:07.454870 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:07.454872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:07.454964 (+ 92us) server_negotiation.cc:407] Connection header received
0928 09:05:07.455020 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:07.455021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:07.455035 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:07.455086 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:07.455384 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.455896 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.456588 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:07.456766 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:07.456817 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:07.459252 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52359: BlockingRecv error: recv got EOF from 127.0.98.3:52359 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 09:05:08 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:08 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:08.206532 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:08.201916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:36005)
0928 09:05:08.202121 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:08.202132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:08.202192 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:08.202316 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:08.202316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:08.202473 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:08.202522 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:08.202524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.203126 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.203126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:08.203706 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:08.203709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.204041 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.204041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:08.204078 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:08.204533 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:08.204538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:08.206418 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":147,"threads_started":1}
W20260928 09:05:08.206584 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:08.201989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36005 (local address 127.0.98.62:39473)
0928 09:05:08.202144 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:08.202146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:08.202181 (+ 35us) server_negotiation.cc:407] Connection header received
0928 09:05:08.202233 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:08.202234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:08.202244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:08.202304 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:08.202579 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.203072 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.203779 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.203970 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.204022 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:08.206511 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36005: BlockingRecv error: recv got EOF from 127.0.98.1:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
Sep 28 09:05:08 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:08 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:08.319953 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:08.315362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:44179)
0928 09:05:08.315536 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:08.315550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:08.315642 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:08.315806 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:08.315807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:08.315982 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:08.316035 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:08.316037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.316688 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.316689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:08.317234 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:08.317236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.317668 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.317669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:08.317726 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:08.318150 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:08.318155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:08.319856 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
W20260928 09:05:08.320005 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:08.315451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44179 (local address 127.0.98.62:39473)
0928 09:05:08.315547 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:08.315550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:08.315621 (+ 71us) server_negotiation.cc:407] Connection header received
0928 09:05:08.315709 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:08.315710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:08.315729 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:08.315785 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:08.316107 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.316607 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.317328 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.317532 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.317670 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:08.319941 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44179: BlockingRecv error: recv got EOF from 127.0.98.2:44179 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 09:05:08 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:08 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:08.464494 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:08.459767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:35835)
0928 09:05:08.459939 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:08.459951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:08.460049 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:08.460196 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:08.460197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:08.460379 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:08.460422 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:08.460424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.461121 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.461123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:08.461667 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:08.461669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.462033 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.462034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:08.462092 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:08.462542 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:08.462547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:08.464410 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
W20260928 09:05:08.464628 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:08.459854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35835 (local address 127.0.98.62:39473)
0928 09:05:08.459925 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:08.459928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:08.460030 (+ 102us) server_negotiation.cc:407] Connection header received
0928 09:05:08.460106 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:08.460107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:08.460121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:08.460188 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:08.460515 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.461024 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.461751 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:08.461940 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:08.462034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:08.464529 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35835: BlockingRecv error: recv got EOF from 127.0.98.3:35835 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 09:05:09 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:09 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:09.211655 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:09.207027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:53345)
0928 09:05:09.207161 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:09.207172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:09.207224 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:09.207350 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:09.207350 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:09.207532 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:09.207580 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:09.207581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.208192 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.208192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:09.208718 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:09.208720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.209108 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.209109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:09.209170 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:09.209824 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:09.209829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:09.211598 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260928 09:05:09.211791 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:09.207085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53345 (local address 127.0.98.62:39473)
0928 09:05:09.207219 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:09.207221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:09.207230 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:05:09.207269 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:09.207269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:09.207280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:09.207336 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:09.207643 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.208131 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.208783 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.209020 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.209089 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:09.211685 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53345: BlockingRecv error: recv got EOF from 127.0.98.1:53345 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
Sep 28 09:05:09 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:09 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:09.324759 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:09.320484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:57427)
0928 09:05:09.320640 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:09.320651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:09.320716 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:09.320828 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:09.320829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:09.320987 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:09.321039 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:09.321041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.321698 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.321700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:09.322232 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:09.322234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.322538 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.322539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:09.322576 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:09.322990 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:09.322994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:09.324699 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
W20260928 09:05:09.324833 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:09.320556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57427 (local address 127.0.98.62:39473)
0928 09:05:09.320627 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:09.320629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:09.320714 (+ 85us) server_negotiation.cc:407] Connection header received
0928 09:05:09.320762 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:09.320763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:09.320773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:09.320824 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:09.321092 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.321633 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.322300 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.322479 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.322528 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:09.324786 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57427: BlockingRecv error: recv got EOF from 127.0.98.2:57427 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 09:05:09 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:09 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:09.469920 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:09.465025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:42053)
0928 09:05:09.465165 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:09.465177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:09.465239 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:09.465497 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:09.465498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:09.465779 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:09.465838 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:09.465841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.466505 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.466506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:09.467091 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:09.467094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.467480 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.467481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:09.467543 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:09.467958 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:09.467962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:09.469856 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
W20260928 09:05:09.470064 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:09.465315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42053 (local address 127.0.98.62:39473)
0928 09:05:09.465385 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:09.465387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:09.465394 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:05:09.465414 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:09.465415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:09.465427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:09.465490 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:09.465904 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.466423 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.467192 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:09.467413 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:09.467465 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:09.469943 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42053: BlockingRecv error: recv got EOF from 127.0.98.3:42053 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 09:05:10 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:10 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:10.224712 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:10.212204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:60025)
0928 09:05:10.212337 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:10.212353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:10.212422 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:10.212570 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:10.212571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:10.212730 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:10.212779 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:10.212780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.213689 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.213690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:10.214216 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:10.214217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.214548 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.214548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:10.214587 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:10.222816 (+ 8229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:10.222823 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:10.224651 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260928 09:05:10.224893 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:10.212300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60025 (local address 127.0.98.62:39473)
0928 09:05:10.212408 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:10.212410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:10.212420 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:05:10.212475 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:10.212475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:10.212486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:10.212557 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:10.213021 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.213604 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.214291 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.214486 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.214540 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:10.224796 (+ 10256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60025: BlockingRecv error: recv got EOF from 127.0.98.1:60025 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
Sep 28 09:05:10 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:10 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:10.330566 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:10.325508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:34701)
0928 09:05:10.325748 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:10.325759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:10.325925 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:10.326077 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:10.326078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:10.326272 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:10.326341 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:10.326343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.327025 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.327026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:10.327565 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:10.327566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.327941 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.327942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:10.328004 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:10.328491 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:10.328495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:10.330503 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
W20260928 09:05:10.330747 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:10.325588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34701 (local address 127.0.98.62:39473)
0928 09:05:10.325664 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:10.325667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:10.325908 (+ 241us) server_negotiation.cc:407] Connection header received
0928 09:05:10.326000 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:10.326001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:10.326014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:10.326061 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:10.326422 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.326938 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.327651 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.327858 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.327940 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:10.330669 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34701: BlockingRecv error: recv got EOF from 127.0.98.2:34701 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 09:05:10 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:10 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:10.475083 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:10.470344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:40275)
0928 09:05:10.470512 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:10.470525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:10.470614 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:10.470780 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:10.470781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:10.470960 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:10.471053 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:10.471054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.471746 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.471747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:10.472275 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:10.472277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.472656 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.472657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:10.472717 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:10.473191 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:10.473196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:10.475024 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
W20260928 09:05:10.475253 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:10.470427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40275 (local address 127.0.98.62:39473)
0928 09:05:10.470508 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:10.470511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:10.470594 (+ 83us) server_negotiation.cc:407] Connection header received
0928 09:05:10.470696 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:10.470697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:10.470715 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:10.470774 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:10.471140 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.471662 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.472363 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:10.472564 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:10.472647 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:10.475139 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40275: BlockingRecv error: recv got EOF from 127.0.98.3:40275 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 09:05:11 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:11 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:11.229975 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:11.225225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:43225)
0928 09:05:11.225362 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:11.225375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:11.225439 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:11.225567 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:11.225568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:11.225724 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:11.225771 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:11.225773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.226460 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.226461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:11.227006 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:11.227008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.227370 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.227376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:11.227424 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:11.227966 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:11.227971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:11.229914 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260928 09:05:11.230167 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:11.225300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43225 (local address 127.0.98.62:39473)
0928 09:05:11.225432 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:11.225434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:11.225449 (+ 15us) server_negotiation.cc:407] Connection header received
0928 09:05:11.225475 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:11.225476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:11.225487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:11.225590 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:11.225841 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.226354 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.227085 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.227282 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.227339 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:11.230096 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43225: BlockingRecv error: recv got EOF from 127.0.98.1:43225 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":47,"threads_started":1}
Sep 28 09:05:11 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:11 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:11.335847 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:11.331137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:50713)
0928 09:05:11.331300 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:11.331312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:11.331405 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:11.331543 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:11.331544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:11.331706 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:11.331755 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:11.331757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.332399 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.332400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:11.332910 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:11.332912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.333241 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.333241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:11.333278 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:11.333767 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:11.333771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:11.335732 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
W20260928 09:05:11.335889 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:11.331266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50713 (local address 127.0.98.62:39473)
0928 09:05:11.331340 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:11.331342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:11.331404 (+ 62us) server_negotiation.cc:407] Connection header received
0928 09:05:11.331475 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:11.331476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:11.331488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:11.331532 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:11.331827 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.332329 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.332987 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.333177 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.333233 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:11.335825 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50713: BlockingRecv error: recv got EOF from 127.0.98.2:50713 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 09:05:11 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:11 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:11.480475 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:11.475553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:50457)
0928 09:05:11.475693 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:11.475705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:11.475796 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:11.475961 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:11.475962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:11.476144 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:11.476189 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:11.476191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.477168 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.477170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:11.477744 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:11.477746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.478110 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.478111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:11.478168 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:11.478651 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:11.478656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:11.480371 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
W20260928 09:05:11.480507 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:11.475611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50457 (local address 127.0.98.62:39473)
0928 09:05:11.475656 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:11.475659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:11.475776 (+ 117us) server_negotiation.cc:407] Connection header received
0928 09:05:11.475864 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:11.475864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:11.475879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:11.475942 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:11.476337 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.477087 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.477829 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:11.478029 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:11.478110 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:11.480446 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50457: BlockingRecv error: recv got EOF from 127.0.98.3:50457 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 09:05:12 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:12 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:12.235023 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:12.230430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:40367)
0928 09:05:12.230566 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:12.230579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:12.230639 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:12.230834 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:12.230835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:12.231002 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:12.231050 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:12.231051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.231676 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.231677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:12.232235 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:12.232237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.232586 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.232587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:12.232637 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:12.233194 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:12.233200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:12.234961 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260928 09:05:12.235198 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:12.230601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40367 (local address 127.0.98.62:39473)
0928 09:05:12.230716 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:12.230718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:12.230725 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:05:12.230746 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:12.230747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:12.230757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:12.230820 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:12.231111 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.231607 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.232309 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.232498 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.232559 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:12.235121 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40367: BlockingRecv error: recv got EOF from 127.0.98.1:40367 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
Sep 28 09:05:12 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:12 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:12.341504 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:12.336514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:38461)
0928 09:05:12.336705 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:12.336720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:12.336785 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:12.336915 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:12.336916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:12.337080 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:12.337129 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:12.337130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.337786 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.337786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:12.338328 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:12.338330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.338709 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.338710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:12.338761 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:12.339400 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:12.339405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:12.341443 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
W20260928 09:05:12.341768 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:12.336637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38461 (local address 127.0.98.62:39473)
0928 09:05:12.336682 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:12.336685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:12.336786 (+ 101us) server_negotiation.cc:407] Connection header received
0928 09:05:12.336846 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:12.336847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:12.336857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:12.336903 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:12.337185 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.337709 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.338411 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.338609 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.338668 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:12.341688 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38461: BlockingRecv error: recv got EOF from 127.0.98.2:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 09:05:12 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:12 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:12.485834 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:12.480955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:57931)
0928 09:05:12.481130 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:12.481143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:12.481232 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:12.481395 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:12.481396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:12.481656 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:12.481704 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:12.481706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.482401 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.482402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:12.482936 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:12.482938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.483320 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.483321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:12.483380 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:12.483876 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:12.483882 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:12.485721 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
W20260928 09:05:12.485872 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:12.481037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57931 (local address 127.0.98.62:39473)
0928 09:05:12.481129 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:12.481131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:12.481214 (+ 83us) server_negotiation.cc:407] Connection header received
0928 09:05:12.481303 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:12.481304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:12.481320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:12.481390 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:12.481787 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.482303 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.483034 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:12.483232 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:12.483319 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:12.485805 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57931: BlockingRecv error: recv got EOF from 127.0.98.3:57931 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 09:05:13 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:13 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:13.240543 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:13.235597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:51491)
0928 09:05:13.235743 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:13.235762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:13.235817 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:13.236002 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:13.236003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:13.236168 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:13.236215 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:13.236216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.236887 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.236888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:13.237453 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:13.237455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.237968 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.237970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:13.238055 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:13.238727 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:13.238732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:13.240458 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260928 09:05:13.240619 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:13.235720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51491 (local address 127.0.98.62:39473)
0928 09:05:13.235850 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:13.235852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:13.235862 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:05:13.235898 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:13.235899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:13.235909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:13.235979 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:13.236286 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.236796 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.237567 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.237762 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.237822 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:13.240565 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51491: BlockingRecv error: recv got EOF from 127.0.98.1:51491 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
Sep 28 09:05:13 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:13 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:13.347106 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:13.342151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:59227)
0928 09:05:13.342305 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:13.342318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:13.342417 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:13.342560 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:13.342561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:13.342741 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:13.342787 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:13.342789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.343522 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.343524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:13.344110 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:13.344111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.344517 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.344518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:13.344576 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:13.345046 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:13.345050 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:13.347006 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
W20260928 09:05:13.347268 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:13.342236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59227 (local address 127.0.98.62:39473)
0928 09:05:13.342310 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:13.342312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:13.342397 (+ 85us) server_negotiation.cc:407] Connection header received
0928 09:05:13.342490 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:13.342491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:13.342503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:13.342557 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:13.342882 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.343433 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.344203 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.344426 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.344515 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:13.347187 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59227: BlockingRecv error: recv got EOF from 127.0.98.2:59227 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 09:05:13 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:13 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:13.491295 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:13.486363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:49993)
0928 09:05:13.486530 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:13.486541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:13.486609 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:13.486739 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:13.486739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:13.486906 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:13.486952 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:13.486953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.487690 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.487691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:13.488236 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:13.488238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.488606 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.488607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:13.488655 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:13.489152 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:13.489157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:13.491238 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
W20260928 09:05:13.491429 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:13.486482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49993 (local address 127.0.98.62:39473)
0928 09:05:13.486530 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:13.486532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:13.486608 (+ 76us) server_negotiation.cc:407] Connection header received
0928 09:05:13.486670 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:13.486670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:13.486683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:13.486725 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:13.487042 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.487577 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.488319 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:13.488521 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:13.488577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:13.491347 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49993: BlockingRecv error: recv got EOF from 127.0.98.3:49993 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 09:05:14 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:14 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:14.246373 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.241241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:44907)
0928 09:05:14.241422 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:14.241433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:14.241482 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:14.241668 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:14.241669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:14.241878 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:14.241927 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.241929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.242670 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.242671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.243292 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.243294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.243693 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.243694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.243737 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.244217 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:14.244222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:14.246244 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
W20260928 09:05:14.246424 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:14.241339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44907 (local address 127.0.98.62:39473)
0928 09:05:14.241475 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:14.241477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:14.241489 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:05:14.241522 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:14.241523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:14.241533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:14.241626 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:14.242033 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.242596 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.243370 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.243627 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.243686 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.246335 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44907: BlockingRecv error: recv got EOF from 127.0.98.1:44907 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Sep 28 09:05:14 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:05:14 dist-test-slave-smns krb5kdc[8535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:05:14.352465 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.347695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:34263)
0928 09:05:14.347853 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:14.347865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:14.347947 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:14.348100 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:14.348101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:14.348261 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:14.348309 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.348311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.349015 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.349015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.349587 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.349589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.349941 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.349941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.349979 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.350427 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:14.350432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:14.352406 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":52,"threads_started":1}
W20260928 09:05:14.352627 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:14.347815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34263 (local address 127.0.98.62:39473)
0928 09:05:14.347864 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:14.347866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:14.347929 (+ 63us) server_negotiation.cc:407] Connection header received
0928 09:05:14.348006 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:14.348007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:14.348018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:14.348065 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:14.348419 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.348939 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.349674 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.349870 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.349930 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.352518 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34263: BlockingRecv error: recv got EOF from 127.0.98.2:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:05:14.407694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9118
I20260928 09:05:14.412107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9270
I20260928 09:05:14.416669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9403
I20260928 09:05:14.421185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9048
2026-09-28T09:05:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17446 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T09:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:05:14Z Disabled control of system clock
I20260928 09:05:14.448827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39445
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 09:05:14.530835 9600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:14.531054 9600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:14.531075 9600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:14.532300 9600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:05:14.532344 9600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:14.532357 9600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:14.532370 9600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:05:14.532425 9600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:05:14.533896 9600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:14.534171 9600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:14.534482 9600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:14.539474 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:14.539537 9608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:14.539572 9600 server_base.cc:1034] running on GCE node
W20260928 09:05:14.539620 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:14.540226 9600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:14.540555 9600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:14.541702 9600 hybrid_clock.cc:648] HybridClock initialized: now 1790586314541678 us; error 43 us; skew 500 ppm
I20260928 09:05:14.542940 9600 webserver.cc:460] Webserver started at http://127.0.98.62:38279/ using document root <none> and password file <none>
I20260928 09:05:14.543215 9600 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:14.543265 9600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:14.543438 9600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:14.544538 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "a5d4f736084148c58df0513cb93d8f60"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "9d5e1c55ec71171950ecfcf708d681af"
server_key_iv: "40acf0c169e9f1e74c7277f1cf1b78e5"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.544914 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5d4f736084148c58df0513cb93d8f60"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "9d5e1c55ec71171950ecfcf708d681af"
server_key_iv: "40acf0c169e9f1e74c7277f1cf1b78e5"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.546456 9600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:05:14.547241 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.547425 9600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:05:14.547533 9600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "a5d4f736084148c58df0513cb93d8f60"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "9d5e1c55ec71171950ecfcf708d681af"
server_key_iv: "40acf0c169e9f1e74c7277f1cf1b78e5"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.547619 9600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:14.572679 9600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:14.573026 9600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:14.573194 9600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:14.577258 9600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39445
I20260928 09:05:14.577298 9666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39445 every 8 connection(s)
I20260928 09:05:14.577679 9600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:05:14.578474 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:05:14.581095 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60: Bootstrap starting.
I20260928 09:05:14.581738 9667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60: Neither blocks nor log segments found. Creating new log.
I20260928 09:05:14.581988 9667 log.cc:824] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60: Log is configured to *not* fsync() on all Append() calls
I20260928 09:05:14.582633 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60: No bootstrap required, opened a new log
I20260928 09:05:14.583508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9600
I20260928 09:05:14.583659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:05:14.583878 392 external_mini_cluster.cc:1428] Setting key b774367fc65b3d337ac6d6dd22fcab85
I20260928 09:05:14.584486 9667 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d4f736084148c58df0513cb93d8f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39445 } }
I20260928 09:05:14.584698 9667 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:05:14.584750 9667 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d4f736084148c58df0513cb93d8f60, State: Initialized, Role: FOLLOWER
I20260928 09:05:14.584884 9667 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d4f736084148c58df0513cb93d8f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39445 } }
I20260928 09:05:14.584985 9667 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:05:14.585028 9667 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:05:14.585076 9667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:05:14.585736 9667 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d4f736084148c58df0513cb93d8f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39445 } }
I20260928 09:05:14.585841 9667 leader_election.cc:302] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5d4f736084148c58df0513cb93d8f60; no voters:
I20260928 09:05:14.585999 9667 leader_election.cc:288] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:05:14.586230 9672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:05:14.586252 9667 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:05:14.586429 9672 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [term 1 LEADER]: Becoming Leader. State: Replica: a5d4f736084148c58df0513cb93d8f60, State: Running, Role: LEADER
I20260928 09:05:14.586510 9672 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d4f736084148c58df0513cb93d8f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39445 } }
I20260928 09:05:14.586944 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5d4f736084148c58df0513cb93d8f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d4f736084148c58df0513cb93d8f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39445 } } }
I20260928 09:05:14.586926 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5d4f736084148c58df0513cb93d8f60. Latest consensus state: current_term: 1 leader_uuid: "a5d4f736084148c58df0513cb93d8f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d4f736084148c58df0513cb93d8f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39445 } } }
I20260928 09:05:14.587021 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [sys.catalog]: This master's current role is: LEADER
I20260928 09:05:14.587036 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60 [sys.catalog]: This master's current role is: LEADER
I20260928 09:05:14.587347 9679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:05:14.587971 9679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:05:14.589838 9679 catalog_manager.cc:1347] Generated new cluster ID: a34723ef5b9d4a5a8201abebfb780806
I20260928 09:05:14.589898 9679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:05:14.590401 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.584482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.0.98.62:39445)
0928 09:05:14.584697 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:14.584704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:14.584753 (+ 49us) server_negotiation.cc:407] Connection header received
0928 09:05:14.584880 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:14.584886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:14.584940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:14.585151 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:14.585485 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.586980 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.588166 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.588724 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.588861 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.589588 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:14.589592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:14.589592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:14.589811 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:14.589854 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:14.589869 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:14.589934 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:14.590126 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 09:05:14.590199 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260928 09:05:14.604249 9679 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:05:14.604976 9679 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:05:14.609581 9679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5d4f736084148c58df0513cb93d8f60: Generated new TSK 0
I20260928 09:05:14.609795 9679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:05:14.614382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39445
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 09:05:14.704044 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:14.704293 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:14.704330 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:14.705536 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:14.705647 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:14.705691 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:05:14.707073 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:14.707347 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:14.707650 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:14.712605 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:14.712661 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:14.712591 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:14.712810 9691 server_base.cc:1034] running on GCE node
I20260928 09:05:14.713088 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:14.713335 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:14.714491 9691 hybrid_clock.cc:648] HybridClock initialized: now 1790586314714478 us; error 33 us; skew 500 ppm
I20260928 09:05:14.715776 9691 webserver.cc:460] Webserver started at http://127.0.98.1:44569/ using document root <none> and password file <none>
I20260928 09:05:14.716044 9691 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:14.716115 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:14.716260 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:14.717334 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "c00955709e0d4813b9e2d440a216b2ff"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "70b8be171f94ef79b498662eafde7da0"
server_key_iv: "ed24bc76ab55c561d6151dd044a3d7e0"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.717726 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c00955709e0d4813b9e2d440a216b2ff"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "70b8be171f94ef79b498662eafde7da0"
server_key_iv: "ed24bc76ab55c561d6151dd044a3d7e0"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.719182 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:05:14.720124 9705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.720353 9691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:05:14.720469 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "c00955709e0d4813b9e2d440a216b2ff"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "70b8be171f94ef79b498662eafde7da0"
server_key_iv: "ed24bc76ab55c561d6151dd044a3d7e0"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.720563 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:14.748184 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:14.748554 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:14.748759 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:14.749110 9691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:14.749661 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:14.749717 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.749756 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:14.749787 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.756361 9691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40775
I20260928 09:05:14.756405 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40775 every 8 connection(s)
I20260928 09:05:14.756770 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:05:14.760015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9691
I20260928 09:05:14.760130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:05:14.760353 392 external_mini_cluster.cc:1428] Setting key 5a92943d35bec5539eb24c0485f4578a
I20260928 09:05:14.761375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39445
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 09:05:14.762462 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.757578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47535 (local address 127.0.98.62:39445)
0928 09:05:14.757637 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:14.757639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:14.757870 (+ 231us) server_negotiation.cc:407] Connection header received
0928 09:05:14.758394 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:14.758395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:14.758408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:14.758474 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:14.759110 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.759677 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.760841 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.761082 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.761175 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.762151 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:14.762152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:14.762153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:14.762158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:14.762172 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:14.762178 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:14.762256 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:14.762342 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 09:05:14.762382 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 09:05:14.762670 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.757549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39445 (local address 127.0.98.1:47535)
0928 09:05:14.757742 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:14.757776 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:14.758294 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:14.758530 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:14.758534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:14.758542 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:14.759030 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.759034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.759787 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.759789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.760726 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.760729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.761184 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.761185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.761321 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.761991 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:05:14.761992 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:14.762021 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:05:14.762044 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:14.762259 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:14.762265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:14.762268 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:05:14.762344 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:05:14.762450 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260928 09:05:14.763257 9819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39445
I20260928 09:05:14.763365 9819 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:14.763551 9819 heartbeater.cc:511] Master 127.0.98.62:39445 requested a full tablet report, sending...
I20260928 09:05:14.764050 9631 ts_manager.cc:194] Registered new tserver with Master: c00955709e0d4813b9e2d440a216b2ff (127.0.98.1:40775)
I20260928 09:05:14.764673 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47535
W20260928 09:05:14.849921 9822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:14.850165 9822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:14.850200 9822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:14.851454 9822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:14.851495 9822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:14.851519 9822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:05:14.852995 9822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:14.853200 9822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:14.853447 9822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:14.858721 9827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:14.858731 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:14.858764 9822 server_base.cc:1034] running on GCE node
W20260928 09:05:14.858731 9828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:14.859157 9822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:14.859404 9822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:14.860575 9822 hybrid_clock.cc:648] HybridClock initialized: now 1790586314860558 us; error 28 us; skew 500 ppm
I20260928 09:05:14.862032 9822 webserver.cc:460] Webserver started at http://127.0.98.2:33333/ using document root <none> and password file <none>
I20260928 09:05:14.862289 9822 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:14.862361 9822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:14.862517 9822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:14.863623 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "7e1e87ac28894472a1a713528f9ade91"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "f0bf6ec7c594d636dc8a77f6040e0d2a"
server_key_iv: "47de141973bbc509f9388e9919927534"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.864004 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7e1e87ac28894472a1a713528f9ade91"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "f0bf6ec7c594d636dc8a77f6040e0d2a"
server_key_iv: "47de141973bbc509f9388e9919927534"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.865381 9822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 09:05:14.866207 9836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.866405 9822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:05:14.866504 9822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "7e1e87ac28894472a1a713528f9ade91"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "f0bf6ec7c594d636dc8a77f6040e0d2a"
server_key_iv: "47de141973bbc509f9388e9919927534"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.866587 9822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:14.880918 9822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:14.881228 9822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:14.881383 9822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:14.881660 9822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:14.881992 9822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:14.882046 9822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.882079 9822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:14.882109 9822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.888899 9822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41743
I20260928 09:05:14.888967 9949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41743 every 8 connection(s)
I20260928 09:05:14.889300 9822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:05:14.894539 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.890113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57441 (local address 127.0.98.62:39445)
0928 09:05:14.890180 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:14.890182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:14.890426 (+ 244us) server_negotiation.cc:407] Connection header received
0928 09:05:14.890902 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:14.890903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:14.890913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:14.890966 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:14.891636 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.892147 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.893101 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.893300 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.893368 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.894306 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:14.894308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:14.894308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:14.894313 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:14.894327 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:14.894330 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:14.894380 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:14.894456 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 09:05:14.894480 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:05:14.894809 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:14.890129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39445 (local address 127.0.98.2:57441)
0928 09:05:14.890302 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:14.890337 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:14.890820 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:14.891025 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:14.891028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:14.891036 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:14.891564 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.891568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.892227 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.892228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.893004 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:14.893006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:14.893382 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:14.893382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:14.893508 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:14.894158 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:05:14.894159 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:14.894188 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:05:14.894212 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:14.894390 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:14.894393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:14.894396 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:05:14.894444 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 09:05:14.894538 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260928 09:05:14.895334 9950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39445
I20260928 09:05:14.895428 9950 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:14.895578 9950 heartbeater.cc:511] Master 127.0.98.62:39445 requested a full tablet report, sending...
I20260928 09:05:14.896000 9631 ts_manager.cc:194] Registered new tserver with Master: 7e1e87ac28894472a1a713528f9ade91 (127.0.98.2:41743)
I20260928 09:05:14.896377 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57441
I20260928 09:05:14.897312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9822
I20260928 09:05:14.897418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:05:14.897644 392 external_mini_cluster.cc:1428] Setting key da9544edefbefc1cf6a05ddc2e242700
I20260928 09:05:14.898551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39445
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 09:05:14.980769 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:14.981031 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:14.981069 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:14.982417 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:14.982494 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:14.982540 9953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:05:14.983938 9953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:14.984232 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:14.984571 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:14.989907 9961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:14.989925 9958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:14.989905 9959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:14.990100 9953 server_base.cc:1034] running on GCE node
I20260928 09:05:14.990314 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:14.990594 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:14.991770 9953 hybrid_clock.cc:648] HybridClock initialized: now 1790586314991772 us; error 44 us; skew 500 ppm
I20260928 09:05:14.993075 9953 webserver.cc:460] Webserver started at http://127.0.98.3:35881/ using document root <none> and password file <none>
I20260928 09:05:14.993331 9953 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:14.993402 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:14.993629 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:14.994717 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "abab7719843c42fd9567fe2cd654a2ad"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "bf074649fa9ccdb24720f7b8f3daee8d"
server_key_iv: "aab742317bc352346dc9e1ae543079e1"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.995090 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "abab7719843c42fd9567fe2cd654a2ad"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "bf074649fa9ccdb24720f7b8f3daee8d"
server_key_iv: "aab742317bc352346dc9e1ae543079e1"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.996416 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:05:14.997162 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:14.997344 9953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:05:14.997464 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "abab7719843c42fd9567fe2cd654a2ad"
format_stamp: "Formatted at 2026-09-28 09:05:14 on dist-test-slave-smns"
server_key: "bf074649fa9ccdb24720f7b8f3daee8d"
server_key_iv: "aab742317bc352346dc9e1ae543079e1"
server_key_version: "encryptionkey@0"
I20260928 09:05:14.997573 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:15.015341 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:15.015705 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:15.015969 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:15.016233 9953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:15.016597 9953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:15.016654 9953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.016697 9953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:15.016722 9953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.022647 9953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44303
I20260928 09:05:15.022686 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44303 every 8 connection(s)
I20260928 09:05:15.023096 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:05:15.023236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 9953
I20260928 09:05:15.023342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:05:15.023639 392 external_mini_cluster.cc:1428] Setting key 952d6c63d0b6e7986d0add92d9f0c4a7
I20260928 09:05:15.028836 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.024317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45383 (local address 127.0.98.62:39445)
0928 09:05:15.024382 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.024384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.024516 (+ 132us) server_negotiation.cc:407] Connection header received
0928 09:05:15.025121 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.025122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.025134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.025187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.025938 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:15.026449 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:15.027471 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:15.027689 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:15.027775 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:15.028561 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:15.028562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:15.028563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:15.028568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:15.028582 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:15.028585 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:15.028629 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:15.028725 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 09:05:15.028754 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:05:15.029160 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.024004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39445 (local address 127.0.98.3:45383)
0928 09:05:15.024387 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:15.024430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:15.025008 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:15.025257 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:15.025261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:15.025271 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:15.025837 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:15.025842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:15.026544 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:15.026545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:15.027368 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:15.027371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:15.027786 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:15.027787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:15.027921 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:15.028423 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:05:15.028423 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:15.028454 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:05:15.028475 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:15.028639 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:15.028643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:15.028646 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:05:15.028729 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:05:15.028828 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":112,"threads_started":1}
I20260928 09:05:15.029606 10081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39445
I20260928 09:05:15.029693 10081 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:15.029884 10081 heartbeater.cc:511] Master 127.0.98.62:39445 requested a full tablet report, sending...
I20260928 09:05:15.030275 9631 ts_manager.cc:194] Registered new tserver with Master: abab7719843c42fd9567fe2cd654a2ad (127.0.98.3:44303)
I20260928 09:05:15.030655 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45383
I20260928 09:05:15.035398 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 09:05:15.037211 9671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:15.036707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.0.98.62:39445)
0928 09:05:15.036777 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.036780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.036867 (+ 87us) server_negotiation.cc:407] Connection header received
0928 09:05:15.036904 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.036904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.036914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.036965 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.037134 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43916: BlockingRecv error: recv got EOF from 127.0.0.1:43916 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:05:15.037232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9691
I20260928 09:05:15.041677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9822
I20260928 09:05:15.045773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9953
I20260928 09:05:15.050252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 9600
2026-09-28T09:05:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (628 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T09:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:05:15Z Disabled control of system clock
I20260928 09:05:15.077229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38797
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:05:15.158336 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:15.158550 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:15.158572 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:15.159783 10092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:05:15.159837 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:15.159849 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:15.159862 10092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:05:15.159919 10092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:05:15.161381 10092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:15.161666 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:15.161972 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:15.167092 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:15.167222 10092 server_base.cc:1034] running on GCE node
W20260928 09:05:15.167066 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.167073 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:15.167550 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:15.167845 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:15.169034 10092 hybrid_clock.cc:648] HybridClock initialized: now 1790586315169020 us; error 67 us; skew 500 ppm
I20260928 09:05:15.170328 10092 webserver.cc:460] Webserver started at http://127.0.98.62:36225/ using document root <none> and password file <none>
I20260928 09:05:15.170598 10092 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:15.170670 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:15.170817 10092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:15.171931 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "04901eab01684738b134b30e55435fd6"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "20dba2b0eb36238ef2a5f8a7946d2d4e"
server_key_iv: "27bf4e27ac2698403d08449908493d76"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.172295 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "04901eab01684738b134b30e55435fd6"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "20dba2b0eb36238ef2a5f8a7946d2d4e"
server_key_iv: "27bf4e27ac2698403d08449908493d76"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.173640 10092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:05:15.174443 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.174613 10092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:05:15.174703 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "04901eab01684738b134b30e55435fd6"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "20dba2b0eb36238ef2a5f8a7946d2d4e"
server_key_iv: "27bf4e27ac2698403d08449908493d76"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.174785 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:15.178236 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:15.178519 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:15.178661 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:15.182700 10092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38797
I20260928 09:05:15.182729 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38797 every 8 connection(s)
I20260928 09:05:15.183066 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:05:15.183674 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:05:15.186460 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6: Bootstrap starting.
I20260928 09:05:15.187004 10159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6: Neither blocks nor log segments found. Creating new log.
I20260928 09:05:15.187242 10159 log.cc:824] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:05:15.187841 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6: No bootstrap required, opened a new log
I20260928 09:05:15.189357 10159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04901eab01684738b134b30e55435fd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260928 09:05:15.189494 10159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:05:15.189517 10159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04901eab01684738b134b30e55435fd6, State: Initialized, Role: FOLLOWER
I20260928 09:05:15.189683 10159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04901eab01684738b134b30e55435fd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260928 09:05:15.189766 10159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:05:15.189810 10159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:05:15.189859 10159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:05:15.190364 10159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04901eab01684738b134b30e55435fd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260928 09:05:15.190480 10159 leader_election.cc:302] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04901eab01684738b134b30e55435fd6; no voters:
I20260928 09:05:15.190644 10159 leader_election.cc:288] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:05:15.190685 10162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:05:15.190860 10162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 04901eab01684738b134b30e55435fd6, State: Running, Role: LEADER
I20260928 09:05:15.190893 10159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:05:15.190945 10162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04901eab01684738b134b30e55435fd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } }
I20260928 09:05:15.191363 10164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04901eab01684738b134b30e55435fd6. Latest consensus state: current_term: 1 leader_uuid: "04901eab01684738b134b30e55435fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04901eab01684738b134b30e55435fd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } } }
I20260928 09:05:15.191650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10092
I20260928 09:05:15.191779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:05:15.191992 392 external_mini_cluster.cc:1428] Setting key 0af1889ac11c09a4d88fd28dbe470764
I20260928 09:05:15.191767 10163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04901eab01684738b134b30e55435fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04901eab01684738b134b30e55435fd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38797 } } }
I20260928 09:05:15.192111 10163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:05:15.192094 10164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:05:15.192436 10178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:05:15.193115 10178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:05:15.194242 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.192516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.98.62:38797)
0928 09:05:15.192700 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.192709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.192742 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:05:15.192854 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.192862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.192915 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.193080 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.193784 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:15.193788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:15.193788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:15.193793 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:15.193849 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:15.193864 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:15.193937 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:15.194015 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 09:05:15.194074 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260928 09:05:15.195048 10178 catalog_manager.cc:1347] Generated new cluster ID: d77dc1ca1cb84ae396f11de8122c0e63
I20260928 09:05:15.195101 10178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:05:15.204753 10178 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:05:15.204910 10178 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:05:15.209409 10178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04901eab01684738b134b30e55435fd6: Generated new TSK 0
I20260928 09:05:15.209630 10178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:05:15.212913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:05:15.295333 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:15.295581 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:15.295620 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:15.296804 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:15.296875 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:15.296921 10183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:05:15.298408 10183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:15.298677 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:15.298985 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:15.304840 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.304834 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.304939 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:15.305230 10183 server_base.cc:1034] running on GCE node
I20260928 09:05:15.305461 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:15.305769 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:15.306928 10183 hybrid_clock.cc:648] HybridClock initialized: now 1790586315306917 us; error 29 us; skew 500 ppm
I20260928 09:05:15.314109 10183 webserver.cc:460] Webserver started at http://127.0.98.1:42809/ using document root <none> and password file <none>
I20260928 09:05:15.314375 10183 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:15.314424 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:15.314553 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:15.315644 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "bf3a3ca481394e30a52665bf016fb7bb"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "5e706773ec06c093315fde7020894703"
server_key_iv: "7768bd4728ab7c3b7ffd6cbcc4c3b251"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.316006 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bf3a3ca481394e30a52665bf016fb7bb"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "5e706773ec06c093315fde7020894703"
server_key_iv: "7768bd4728ab7c3b7ffd6cbcc4c3b251"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.317342 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:05:15.318195 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.318364 10183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:05:15.318460 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "bf3a3ca481394e30a52665bf016fb7bb"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "5e706773ec06c093315fde7020894703"
server_key_iv: "7768bd4728ab7c3b7ffd6cbcc4c3b251"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.318533 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:15.321929 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:15.322217 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:15.322351 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:15.322582 10183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:15.322882 10183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:15.322913 10183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.322963 10183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:15.322999 10183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.328819 10183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44525
I20260928 09:05:15.328860 10310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44525 every 8 connection(s)
I20260928 09:05:15.329174 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:05:15.332305 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.329898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57021 (local address 127.0.98.62:38797)
0928 09:05:15.329958 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.329960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.330211 (+ 251us) server_negotiation.cc:407] Connection header received
0928 09:05:15.330756 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.330758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.330767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.330813 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.332041 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:15.332042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:15.332042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:15.332047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:15.332061 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:15.332066 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:15.332121 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:15.332219 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 09:05:15.332236 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:05:15.332499 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.329897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38797 (local address 127.0.98.1:57021)
0928 09:05:15.330093 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:15.330126 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:15.330636 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:15.330894 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:15.330898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:15.330908 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:15.331873 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:05:15.331874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:15.331901 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:05:15.331929 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:15.332127 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:15.332131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:15.332135 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:05:15.332229 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 09:05:15.332289 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 09:05:15.332938 10311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38797
I20260928 09:05:15.333050 10311 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:15.333201 10311 heartbeater.cc:511] Master 127.0.98.62:38797 requested a full tablet report, sending...
I20260928 09:05:15.333751 10123 ts_manager.cc:194] Registered new tserver with Master: bf3a3ca481394e30a52665bf016fb7bb (127.0.98.1:44525)
I20260928 09:05:15.337990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10183
I20260928 09:05:15.338064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:05:15.338292 392 external_mini_cluster.cc:1428] Setting key 745a4d59c62ceab91b75f45a0aa36d29
I20260928 09:05:15.339231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:05:15.421288 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:15.421509 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:15.421535 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:15.422904 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:15.422953 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:15.422989 10314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:05:15.424288 10314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:15.424506 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:15.424767 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:15.430020 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.430058 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.430058 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:15.430301 10314 server_base.cc:1034] running on GCE node
I20260928 09:05:15.430526 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:15.430764 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:15.432029 10314 hybrid_clock.cc:648] HybridClock initialized: now 1790586315431997 us; error 43 us; skew 500 ppm
I20260928 09:05:15.433326 10314 webserver.cc:460] Webserver started at http://127.0.98.2:41427/ using document root <none> and password file <none>
I20260928 09:05:15.433601 10314 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:15.433650 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:15.433749 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:15.434907 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "7966f0596a354b0c936060153d06db88"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "dbda3a1e1a524c22ec7a56359f0b81a1"
server_key_iv: "5a2a0b098f3ed2007b80d23346565f0d"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.435221 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7966f0596a354b0c936060153d06db88"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "dbda3a1e1a524c22ec7a56359f0b81a1"
server_key_iv: "5a2a0b098f3ed2007b80d23346565f0d"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.436492 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:05:15.437296 10328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.437446 10314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:05:15.437534 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "7966f0596a354b0c936060153d06db88"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "dbda3a1e1a524c22ec7a56359f0b81a1"
server_key_iv: "5a2a0b098f3ed2007b80d23346565f0d"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.437646 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:15.440857 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:15.441064 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:15.441174 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:15.441322 10314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:15.441617 10314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:15.441649 10314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.441670 10314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:15.441684 10314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.447069 10314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35817
I20260928 09:05:15.447140 10441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35817 every 8 connection(s)
I20260928 09:05:15.447476 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:05:15.450237 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.448138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37891 (local address 127.0.98.62:38797)
0928 09:05:15.448202 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.448204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.448437 (+ 233us) server_negotiation.cc:407] Connection header received
0928 09:05:15.448894 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.448895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.448900 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.448943 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.450034 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:15.450035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:15.450035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:15.450040 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:15.450055 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:15.450059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:15.450112 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:15.450172 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 09:05:15.450185 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 09:05:15.450430 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.448139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38797 (local address 127.0.98.2:37891)
0928 09:05:15.448314 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:15.448352 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:15.448798 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:15.448994 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:15.448998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:15.449006 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:15.449895 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:05:15.449896 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:15.449923 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:05:15.449951 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:15.450112 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:15.450115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:15.450117 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:05:15.450169 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:05:15.450215 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 09:05:15.450904 10442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38797
I20260928 09:05:15.451000 10442 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:15.451180 10442 heartbeater.cc:511] Master 127.0.98.62:38797 requested a full tablet report, sending...
I20260928 09:05:15.451532 10123 ts_manager.cc:194] Registered new tserver with Master: 7966f0596a354b0c936060153d06db88 (127.0.98.2:35817)
I20260928 09:05:15.453747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10314
I20260928 09:05:15.453832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:05:15.453989 392 external_mini_cluster.cc:1428] Setting key f1f0103430786608c6507c1fb521ab8b
I20260928 09:05:15.454829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:05:15.535283 10445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:15.535524 10445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:15.535559 10445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:15.536724 10445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:15.536787 10445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:15.536828 10445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:05:15.538261 10445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:15.538533 10445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:15.538832 10445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:15.544036 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.543988 10451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:15.543998 10453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:15.544349 10445 server_base.cc:1034] running on GCE node
I20260928 09:05:15.544574 10445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:15.544901 10445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:15.546088 10445 hybrid_clock.cc:648] HybridClock initialized: now 1790586315546081 us; error 51 us; skew 500 ppm
I20260928 09:05:15.547430 10445 webserver.cc:460] Webserver started at http://127.0.98.3:46187/ using document root <none> and password file <none>
I20260928 09:05:15.547703 10445 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:15.547773 10445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:15.547912 10445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:15.549031 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "66e3229e758f4260a9bc1eb4b5281f15"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "00dfe10809af14d1b94ec33872202bcf"
server_key_iv: "fa754ab41314b3c8da76499e2186d066"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.549391 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66e3229e758f4260a9bc1eb4b5281f15"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "00dfe10809af14d1b94ec33872202bcf"
server_key_iv: "fa754ab41314b3c8da76499e2186d066"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.550767 10445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:05:15.551565 10459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.551740 10445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:05:15.551836 10445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "66e3229e758f4260a9bc1eb4b5281f15"
format_stamp: "Formatted at 2026-09-28 09:05:15 on dist-test-slave-smns"
server_key: "00dfe10809af14d1b94ec33872202bcf"
server_key_iv: "fa754ab41314b3c8da76499e2186d066"
server_key_version: "encryptionkey@0"
I20260928 09:05:15.551919 10445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:15.555373 10445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:15.555642 10445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:15.555780 10445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:15.555974 10445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:15.556281 10445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:15.556331 10445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.556365 10445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:15.556391 10445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:15.561954 10445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38923
I20260928 09:05:15.561995 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38923 every 8 connection(s)
I20260928 09:05:15.562309 10445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:05:15.565368 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.562974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36015 (local address 127.0.98.62:38797)
0928 09:05:15.563039 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.563041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.563274 (+ 233us) server_negotiation.cc:407] Connection header received
0928 09:05:15.563808 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.563810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.563816 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.563866 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.565110 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:15.565111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:15.565112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:05:15.565117 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:15.565133 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:05:15.565138 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:15.565192 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:15.565281 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:05:15.565297 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:05:15.565591 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:15.562962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38797 (local address 127.0.98.3:36015)
0928 09:05:15.563159 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:15.563191 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:15.563691 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:15.563942 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:15.563945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:15.563953 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:15.564942 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:05:15.564942 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:15.564977 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:05:15.565007 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:15.565188 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:15.565192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:15.565196 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:05:15.565291 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 09:05:15.565352 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260928 09:05:15.566082 10573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38797
I20260928 09:05:15.566159 10573 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:15.566321 10573 heartbeater.cc:511] Master 127.0.98.62:38797 requested a full tablet report, sending...
I20260928 09:05:15.566704 10123 ts_manager.cc:194] Registered new tserver with Master: 66e3229e758f4260a9bc1eb4b5281f15 (127.0.98.3:38923)
I20260928 09:05:15.569077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10445
I20260928 09:05:15.569175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:05:15.569365 392 external_mini_cluster.cc:1428] Setting key 2af5cb2223853efb9364e912580a01e5
I20260928 09:05:15.570310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 09:05:15.571991 10180 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:05:15.571560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.0.98.62:38797)
0928 09:05:15.571628 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:15.571630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:15.571667 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:05:15.571734 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:15.571734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:15.571738 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:15.571786 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:15.571925 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34950: BlockingRecv error: recv got EOF from 127.0.0.1:34950 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:05:15.572045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10183
I20260928 09:05:15.577235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10314
I20260928 09:05:15.581774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10445
I20260928 09:05:15.585989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10092
2026-09-28T09:05:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (535 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:05:15 dist-test-slave-smns krb5kdc[10583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:05:15 dist-test-slave-smns krb5kdc[10583](info): set up 2 sockets
Sep 28 09:05:15 dist-test-slave-smns krb5kdc[10583](info): commencing operation
krb5kdc: starting...
W20260928 09:05:16.630632 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:05:16 dist-test-slave-smns krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:05:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:05:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:05:16.756393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:05:16.842056 10596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:16.842275 10596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:16.842300 10596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:16.843572 10596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:05:16.843647 10596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:16.843688 10596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:16.843747 10596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:05:16.843770 10596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:05:16.845343 10596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:16.845651 10596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:16.845901 10596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:16.850806 10602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:16.850863 10601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:16.850931 10596 server_base.cc:1034] running on GCE node
W20260928 09:05:16.850844 10604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:16.851254 10596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:16.851519 10596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:16.852669 10596 hybrid_clock.cc:648] HybridClock initialized: now 1790586316852656 us; error 29 us; skew 500 ppm
Sep 28 09:05:16 dist-test-slave-smns krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:05:16.855238 10596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:05:16.855715 10596 webserver.cc:460] Webserver started at http://127.0.98.62:33603/ using document root <none> and password file <none>
I20260928 09:05:16.855960 10596 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:16.856029 10596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:16.856189 10596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:16.857316 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a"
format_stamp: "Formatted at 2026-09-28 09:05:16 on dist-test-slave-smns"
server_key: "fdb088af8dcb59dd44332645beb780c5"
server_key_iv: "04aeed9962fafe5d4840817a2834b4e1"
server_key_version: "encryptionkey@0"
I20260928 09:05:16.857728 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a"
format_stamp: "Formatted at 2026-09-28 09:05:16 on dist-test-slave-smns"
server_key: "fdb088af8dcb59dd44332645beb780c5"
server_key_iv: "04aeed9962fafe5d4840817a2834b4e1"
server_key_version: "encryptionkey@0"
I20260928 09:05:16.859112 10596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:05:16.859964 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:16.860224 10596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:05:16.860319 10596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a"
format_stamp: "Formatted at 2026-09-28 09:05:16 on dist-test-slave-smns"
server_key: "fdb088af8dcb59dd44332645beb780c5"
server_key_iv: "04aeed9962fafe5d4840817a2834b4e1"
server_key_version: "encryptionkey@0"
I20260928 09:05:16.860414 10596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:16.873935 10596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:16.875808 10596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:16.875979 10596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:16.880151 10596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37595
I20260928 09:05:16.880225 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37595 every 8 connection(s)
I20260928 09:05:16.880584 10596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:05:16.881393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10596
I20260928 09:05:16.881520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:05:16.881776 392 external_mini_cluster.cc:1428] Setting key d79aa285a7e173f76e190c6f949daaef
I20260928 09:05:16.881798 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:05:16.885140 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a: Bootstrap starting.
I20260928 09:05:16.885857 10664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a: Neither blocks nor log segments found. Creating new log.
I20260928 09:05:16.886209 10664 log.cc:824] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a: Log is configured to *not* fsync() on all Append() calls
I20260928 09:05:16.886941 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a: No bootstrap required, opened a new log
Sep 28 09:05:16 dist-test-slave-smns krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:05:16.888896 10664 raft_consensus.cc:348] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260928 09:05:16.889101 10664 raft_consensus.cc:374] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:05:16.889155 10664 raft_consensus.cc:729] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97a1c571b6e4b41b9e977f0c35a4f4a, State: Initialized, Role: FOLLOWER
I20260928 09:05:16.889285 10664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260928 09:05:16.889380 10664 raft_consensus.cc:388] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:05:16.889415 10664 raft_consensus.cc:482] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:05:16.889472 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:05:16.890249 10664 raft_consensus.cc:504] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260928 09:05:16.890355 10664 leader_election.cc:302] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e97a1c571b6e4b41b9e977f0c35a4f4a; no voters:
I20260928 09:05:16.890516 10664 leader_election.cc:288] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:05:16.890591 10669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:05:16.890738 10669 raft_consensus.cc:686] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [term 1 LEADER]: Becoming Leader. State: Replica: e97a1c571b6e4b41b9e977f0c35a4f4a, State: Running, Role: LEADER
I20260928 09:05:16.890776 10664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:05:16.890829 10669 consensus_queue.cc:237] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260928 09:05:16.891162 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97a1c571b6e4b41b9e977f0c35a4f4a. Latest consensus state: current_term: 1 leader_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } } }
I20260928 09:05:16.891254 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [sys.catalog]: This master's current role is: LEADER
I20260928 09:05:16.891463 10676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:05:16.891562 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97a1c571b6e4b41b9e977f0c35a4f4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } } }
I20260928 09:05:16.891676 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a [sys.catalog]: This master's current role is: LEADER
I20260928 09:05:16.891983 10676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:05:16.892827 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:16.882336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.0.98.62:37595)
0928 09:05:16.882537 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:16.882545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:16.882570 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:05:16.882958 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:16.882964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:16.883008 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:16.883127 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:16.883937 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:16.884820 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:16.885897 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:16.886425 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:16.886521 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:16.888784 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:16.888789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:16.888789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:05:16.888801 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:16.890908 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:16.891439 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:16.891442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:16.891442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:16.891569 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:16.891811 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:16.891812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:16.891813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:16.892182 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:16.892346 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:16.892457 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 09:05:16.892608 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260928 09:05:16.894945 10676 catalog_manager.cc:1347] Generated new cluster ID: 9ee93c1712ed4fcf935bf064a4fbce94
I20260928 09:05:16.895048 10676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:05:16.921692 10676 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:05:16.922291 10676 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:05:16.930963 10676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a: Generated new TSK 0
I20260928 09:05:16.931190 10676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:05:16.964632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37595
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:05:17.048172 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:17.048381 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:17.048410 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:17.049729 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:17.049774 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:17.049803 10690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:05:17.051312 10690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:17.051574 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:17.051826 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:17.057157 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:17.057135 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:17.057185 10690 server_base.cc:1034] running on GCE node
W20260928 09:05:17.057135 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:17.057684 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:17.058039 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:17.059178 10690 hybrid_clock.cc:648] HybridClock initialized: now 1790586317059156 us; error 46 us; skew 500 ppm
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:05:17.062266 10690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:05:17.062886 10690 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260928 09:05:17.063162 10690 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:17.063241 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:17.063463 10690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:17.064819 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "5dc191a6a92c4f5fa8cf335de13557aa"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "38284f5747347079ac2d08dcb5e622b1"
server_key_iv: "22e4ba9f911cf8ded37960c3cdc81ed6"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.065219 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5dc191a6a92c4f5fa8cf335de13557aa"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "38284f5747347079ac2d08dcb5e622b1"
server_key_iv: "22e4ba9f911cf8ded37960c3cdc81ed6"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.066685 10690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 09:05:17.067562 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.067736 10690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:05:17.067850 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "5dc191a6a92c4f5fa8cf335de13557aa"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "38284f5747347079ac2d08dcb5e622b1"
server_key_iv: "22e4ba9f911cf8ded37960c3cdc81ed6"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.067939 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:17.083482 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:17.085968 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:17.086138 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:17.086378 10690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:17.086699 10690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:17.086752 10690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.086818 10690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:17.086848 10690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.093501 10690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41603
I20260928 09:05:17.093566 10818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41603 every 8 connection(s)
I20260928 09:05:17.093889 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:05:17.099900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10690
I20260928 09:05:17.100008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:05:17.100271 392 external_mini_cluster.cc:1428] Setting key 1202657d6d1e5a53860722f69fcc089b
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:05:17.105310 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.094661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56253 (local address 127.0.98.62:37595)
0928 09:05:17.094724 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.094726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.094929 (+ 203us) server_negotiation.cc:407] Connection header received
0928 09:05:17.095494 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.095496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.095508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.095551 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:17.096618 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.097479 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.098665 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.099093 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.099179 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.101989 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:17.101990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:17.101991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:05:17.102000 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:17.103691 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.104114 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.104116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.104116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.104175 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.104440 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.104441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.104442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.104564 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:17.104634 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.105105 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.105211 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:05:17.105604 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.094653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37595 (local address 127.0.98.1:56253)
0928 09:05:17.094804 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.094840 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.095378 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.095634 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.095639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.096213 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:17.096491 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.096497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.097624 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.097625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.098569 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.098572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.099204 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.099205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.099358 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.100049 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:17.100058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:17.101880 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:17.103782 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:05:17.103789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:05:17.103791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:05:17.104034 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:05:17.104247 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:05:17.104248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:05:17.104249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:05:17.104372 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:05:17.104637 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:17.104644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:17.104800 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.105081 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.105267 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 09:05:17.106165 10819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37595
I20260928 09:05:17.106262 10819 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:17.106472 10819 heartbeater.cc:511] Master 127.0.98.62:37595 requested a full tablet report, sending...
I20260928 09:05:17.107036 10628 ts_manager.cc:194] Registered new tserver with Master: 5dc191a6a92c4f5fa8cf335de13557aa (127.0.98.1:41603)
I20260928 09:05:17.107756 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56253
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:05:17.128938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37595
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:05:17.211758 10824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:17.211968 10824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:17.211992 10824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:17.213238 10824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:17.213280 10824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:17.213307 10824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:05:17.214926 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:17.215193 10824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:17.215432 10824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:17.220438 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:17.220467 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:17.220438 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:17.220470 10824 server_base.cc:1034] running on GCE node
I20260928 09:05:17.220904 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:17.221225 10824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:17.222375 10824 hybrid_clock.cc:648] HybridClock initialized: now 1790586317222354 us; error 43 us; skew 500 ppm
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:05:17.224984 10824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:05:17.225495 10824 webserver.cc:460] Webserver started at http://127.0.98.2:35097/ using document root <none> and password file <none>
I20260928 09:05:17.225811 10824 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:17.225858 10824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:17.226001 10824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:17.227075 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "4275d62117064a57b6afe3f1a6fa2801"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "5bf2cab21801aa2670ec2939f423b34c"
server_key_iv: "8e91e8eb08e9d8a11199635b9573e91b"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.227437 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4275d62117064a57b6afe3f1a6fa2801"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "5bf2cab21801aa2670ec2939f423b34c"
server_key_iv: "8e91e8eb08e9d8a11199635b9573e91b"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.228806 10824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:05:17.229621 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.229815 10824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:05:17.229913 10824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "4275d62117064a57b6afe3f1a6fa2801"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "5bf2cab21801aa2670ec2939f423b34c"
server_key_iv: "8e91e8eb08e9d8a11199635b9573e91b"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.230001 10824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:17.240312 10824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:17.242219 10824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:17.242381 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:17.242614 10824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:17.242924 10824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:17.242985 10824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.243024 10824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:17.243052 10824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.248909 10824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45389
I20260928 09:05:17.248960 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45389 every 8 connection(s)
I20260928 09:05:17.249279 10824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:05:17.253727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10824
I20260928 09:05:17.253835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:05:17.254038 392 external_mini_cluster.cc:1428] Setting key 71d8e098322b800c5ac60313de099966
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:05:17.259541 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.249996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41005 (local address 127.0.98.62:37595)
0928 09:05:17.250074 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.250077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.250341 (+ 264us) server_negotiation.cc:407] Connection header received
0928 09:05:17.250881 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.250882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.250894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.250951 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:17.251919 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.252436 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.253461 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.253758 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.253844 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.256463 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:17.256465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:17.256466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:05:17.256475 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:17.258000 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.258532 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.258533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.258534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.258585 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.258880 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.258881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.258881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.258984 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:17.259038 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.259387 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.259457 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:05:17.259862 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.249970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37595 (local address 127.0.98.2:41005)
0928 09:05:17.250213 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.250248 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.250773 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.251035 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.251039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.251619 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:17.251815 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.251819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.252547 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.252548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.253363 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.253365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.253853 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.253854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.253977 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.254640 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:17.254651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:17.256368 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:17.258090 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:05:17.258098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:05:17.258100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:05:17.258447 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:05:17.258659 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:05:17.258659 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:05:17.258660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:05:17.258811 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:05:17.259051 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:17.259057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:17.259236 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.259370 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.259565 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":132,"threads_started":1}
I20260928 09:05:17.260444 10953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37595
I20260928 09:05:17.260550 10953 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:17.260756 10953 heartbeater.cc:511] Master 127.0.98.62:37595 requested a full tablet report, sending...
I20260928 09:05:17.261162 10628 ts_manager.cc:194] Registered new tserver with Master: 4275d62117064a57b6afe3f1a6fa2801 (127.0.98.2:45389)
I20260928 09:05:17.261628 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41005
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:05:17.281118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37595
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:05:17.362493 10958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:05:17.362743 10958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:05:17.362782 10958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:05:17.364092 10958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:05:17.364158 10958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:05:17.364205 10958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:05:17.365743 10958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:05:17.366034 10958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:05:17.366271 10958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:05:17.371753 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:17.371857 10958 server_base.cc:1034] running on GCE node
W20260928 09:05:17.371739 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:05:17.371730 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:05:17.372136 10958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:05:17.372404 10958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:05:17.373631 10958 hybrid_clock.cc:648] HybridClock initialized: now 1790586317373623 us; error 35 us; skew 500 ppm
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:05:17.376300 10958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:05:17.376811 10958 webserver.cc:460] Webserver started at http://127.0.98.3:39281/ using document root <none> and password file <none>
I20260928 09:05:17.377072 10958 fs_manager.cc:359] Metadata directory not provided
I20260928 09:05:17.377143 10958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:05:17.377297 10958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:05:17.378458 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "def2a9569215e0d6a0f30406ec68264d"
server_key_iv: "676fac9dc9d276a0e81c762522bef527"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.378830 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "def2a9569215e0d6a0f30406ec68264d"
server_key_iv: "676fac9dc9d276a0e81c762522bef527"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.380220 10958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:05:17.381043 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.381233 10958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:05:17.381340 10958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57"
format_stamp: "Formatted at 2026-09-28 09:05:17 on dist-test-slave-smns"
server_key: "def2a9569215e0d6a0f30406ec68264d"
server_key_iv: "676fac9dc9d276a0e81c762522bef527"
server_key_version: "encryptionkey@0"
I20260928 09:05:17.381466 10958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:05:17.398083 10958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:05:17.400499 10958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:05:17.400664 10958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:05:17.400900 10958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:05:17.401232 10958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:05:17.401299 10958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.401345 10958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:05:17.401369 10958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:05:17.407424 10958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44965
I20260928 09:05:17.407462 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44965 every 8 connection(s)
I20260928 09:05:17.407799 10958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:05:17.415910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 10958
I20260928 09:05:17.416039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:05:17.416237 392 external_mini_cluster.cc:1428] Setting key f4d8837cb83fcafc8ad92e2cc6420c67
I20260928 09:05:17.417286 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.408555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35259 (local address 127.0.98.62:37595)
0928 09:05:17.408604 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.408606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.408787 (+ 181us) server_negotiation.cc:407] Connection header received
0928 09:05:17.409326 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.409327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.409341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.409387 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:17.410356 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.410878 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.411894 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.412093 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.412182 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.414420 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:17.414422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:17.414423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:05:17.414430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:17.415652 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.416094 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.416096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.416096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.416155 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.416489 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.416491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.416491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.416681 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:17.416757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.417087 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.417194 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:05:17.417495 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.408521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37595 (local address 127.0.98.3:35259)
0928 09:05:17.408662 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.408698 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.409214 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.409461 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.409469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.410056 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:05:17.410251 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.410255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.410977 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.410978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.411794 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.411796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.412190 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.412191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.412316 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.412812 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:05:17.412821 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:05:17.414304 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:05:17.415759 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:05:17.415765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:05:17.415768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:05:17.415990 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:05:17.416240 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:05:17.416240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:05:17.416241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:05:17.416393 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:05:17.416788 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:05:17.416794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:05:17.416949 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.417046 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.417237 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260928 09:05:17.417943 11087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37595
I20260928 09:05:17.418056 11087 heartbeater.cc:464] Registering TS with master...
I20260928 09:05:17.418247 11087 heartbeater.cc:511] Master 127.0.98.62:37595 requested a full tablet report, sending...
I20260928 09:05:17.418606 10628 ts_manager.cc:194] Registered new tserver with Master: 6f3dfd95fbd94f5b90b89bfcd43e3d57 (127.0.98.3:44965)
I20260928 09:05:17.418987 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35259
I20260928 09:05:17.428097 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:05:17.435520 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.429620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.0.98.62:37595)
0928 09:05:17.429682 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.429684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.429695 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:05:17.429742 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.429743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.429751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.429808 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:05:17.430366 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.430877 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.431679 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.431865 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.431921 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.433197 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:05:17.433199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:05:17.433199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:05:17.433223 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:05:17.434542 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.434857 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.434858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.434858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.434907 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:05:17.435076 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:05:17.435077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:05:17.435077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:05:17.435214 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:05:17.435292 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.435393 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.435454 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:05:17.437042 10628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:05:17.438645 10628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:05:17.445376 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.440982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44965 (local address 127.0.0.1:53230)
0928 09:05:17.441148 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.441187 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.441255 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.441611 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.441614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.441617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:05:17.441686 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.441688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.443021 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.443022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.444163 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.444166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.445089 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.445090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.445150 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.445162 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.445224 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.445296 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":43,"thread_start_us":61,"threads_started":1}
I20260928 09:05:17.445391 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.441072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.0.98.3:44965)
0928 09:05:17.441404 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.441409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.441419 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:05:17.441465 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.441467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.441479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.441605 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:05:17.441744 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.442905 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.444260 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.444988 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.445078 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.445141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.445258 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.445299 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":70,"threads_started":1}
I20260928 09:05:17.445536 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.441211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41603 (local address 127.0.0.1:48646)
0928 09:05:17.441384 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.441395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.441445 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.442188 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.442188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.442190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:05:17.442233 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.442234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.443065 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.443066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.444356 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.444359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.445298 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.445299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.445387 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.445391 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.445448 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.445488 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":45,"thread_start_us":80,"threads_started":1}
I20260928 09:05:17.446233 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.441324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.0.98.1:41603)
0928 09:05:17.442054 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.442058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.442065 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:05:17.442094 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.442095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.442107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.442190 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:05:17.442291 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.442966 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.444449 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.445193 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.445283 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.445335 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.446083 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.446137 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":79,"threads_started":1}
I20260928 09:05:17.446261 11021 tablet_service.cc:1467] Processing CreateTablet for tablet d34ee887149d47c5a688b9b7bf05656d (DEFAULT_TABLE table=test-table [id=4da022ae25e84fda9cbafd50026e0a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:05:17.446496 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.440991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45389 (local address 127.0.0.1:42792)
0928 09:05:17.441327 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.441337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.441383 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.441866 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.441867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.441869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:05:17.441917 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.441919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.443157 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.443158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.444357 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.444359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.446258 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.446259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.446322 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.446326 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.446384 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.446424 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":57,"thread_start_us":97,"threads_started":1}
I20260928 09:05:17.446528 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.441113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.0.98.2:45389)
0928 09:05:17.441612 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.441617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.441628 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:05:17.441670 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.441672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.441688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.441761 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:05:17.441981 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.443015 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.445373 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.446194 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.446260 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.446313 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.446397 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.446435 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":70,"threads_started":1}
I20260928 09:05:17.446663 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34ee887149d47c5a688b9b7bf05656d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:05:17.447095 10887 tablet_service.cc:1467] Processing CreateTablet for tablet d34ee887149d47c5a688b9b7bf05656d (DEFAULT_TABLE table=test-table [id=4da022ae25e84fda9cbafd50026e0a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:05:17.447095 10753 tablet_service.cc:1467] Processing CreateTablet for tablet d34ee887149d47c5a688b9b7bf05656d (DEFAULT_TABLE table=test-table [id=4da022ae25e84fda9cbafd50026e0a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:05:17.447420 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34ee887149d47c5a688b9b7bf05656d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:05:17.447420 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34ee887149d47c5a688b9b7bf05656d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:05:17.449568 11104 tablet_bootstrap.cc:492] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Bootstrap starting.
I20260928 09:05:17.450202 11104 tablet_bootstrap.cc:654] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Neither blocks nor log segments found. Creating new log.
I20260928 09:05:17.450706 11106 tablet_bootstrap.cc:492] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Bootstrap starting.
I20260928 09:05:17.450824 11104 log.cc:824] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Log is configured to *not* fsync() on all Append() calls
I20260928 09:05:17.450937 11105 tablet_bootstrap.cc:492] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: Bootstrap starting.
I20260928 09:05:17.451592 11105 tablet_bootstrap.cc:654] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: Neither blocks nor log segments found. Creating new log.
I20260928 09:05:17.451671 11104 tablet_bootstrap.cc:492] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: No bootstrap required, opened a new log
I20260928 09:05:17.451735 11104 ts_tablet_manager.cc:1397] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:05:17.452149 11105 log.cc:824] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: Log is configured to *not* fsync() on all Append() calls
I20260928 09:05:17.452840 11105 tablet_bootstrap.cc:492] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: No bootstrap required, opened a new log
I20260928 09:05:17.452925 11105 ts_tablet_manager.cc:1397] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:05:17.452988 11106 tablet_bootstrap.cc:654] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Neither blocks nor log segments found. Creating new log.
I20260928 09:05:17.453531 11106 log.cc:824] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Log is configured to *not* fsync() on all Append() calls
I20260928 09:05:17.454035 11104 raft_consensus.cc:348] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.454224 11104 raft_consensus.cc:374] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:05:17.454329 11106 tablet_bootstrap.cc:492] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: No bootstrap required, opened a new log
I20260928 09:05:17.454350 11104 raft_consensus.cc:729] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3dfd95fbd94f5b90b89bfcd43e3d57, State: Initialized, Role: FOLLOWER
I20260928 09:05:17.454384 11106 ts_tablet_manager.cc:1397] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 09:05:17.454507 11104 consensus_queue.cc:260] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.454784 11104 ts_tablet_manager.cc:1428] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:05:17.454803 11087 heartbeater.cc:503] Master 127.0.98.62:37595 was elected leader, sending a full tablet report...
I20260928 09:05:17.455060 11105 raft_consensus.cc:348] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.455262 11105 raft_consensus.cc:374] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:05:17.455309 11105 raft_consensus.cc:729] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc191a6a92c4f5fa8cf335de13557aa, State: Initialized, Role: FOLLOWER
I20260928 09:05:17.455432 11105 consensus_queue.cc:260] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.455739 11105 ts_tablet_manager.cc:1428] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:05:17.455762 10819 heartbeater.cc:503] Master 127.0.98.62:37595 was elected leader, sending a full tablet report...
I20260928 09:05:17.456729 11106 raft_consensus.cc:348] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.456908 11106 raft_consensus.cc:374] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:05:17.456954 11106 raft_consensus.cc:729] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4275d62117064a57b6afe3f1a6fa2801, State: Initialized, Role: FOLLOWER
I20260928 09:05:17.457068 11106 consensus_queue.cc:260] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.457329 11106 ts_tablet_manager.cc:1428] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:05:17.457361 10953 heartbeater.cc:503] Master 127.0.98.62:37595 was elected leader, sending a full tablet report...
W20260928 09:05:17.499876 10954 tablet.cc:2369] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:05:17.568943 11110 raft_consensus.cc:482] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:05:17.569056 11110 raft_consensus.cc:504] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.569479 11110 leader_election.cc:288] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4275d62117064a57b6afe3f1a6fa2801 (127.0.98.2:45389), 5dc191a6a92c4f5fa8cf335de13557aa (127.0.98.1:41603)
I20260928 09:05:17.573783 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.569941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49925 (local address 127.0.98.2:45389)
0928 09:05:17.570009 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.570011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.570019 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:05:17.570049 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.570050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.570084 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.570145 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:05:17.570337 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.571114 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.572718 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.573391 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.573458 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.573469 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.573660 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.573706 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:05:17.573783 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.570054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51999 (local address 127.0.98.1:41603)
0928 09:05:17.570117 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.570120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.570227 (+ 107us) server_negotiation.cc:407] Connection header received
0928 09:05:17.570355 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.570356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.570370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.570429 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:05:17.570571 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.571263 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.572717 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.573296 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.573440 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.573455 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.573665 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.573705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:05:17.573807 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.569784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45389 (local address 127.0.98.3:49925)
0928 09:05:17.569852 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.569867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.569936 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.570180 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.570181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.570183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:05:17.570263 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.570265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.571222 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.571223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.572589 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.572594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.573479 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.573479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.573574 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.573576 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.573647 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.573705 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":93}
I20260928 09:05:17.573814 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.569918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41603 (local address 127.0.98.3:51999)
0928 09:05:17.570147 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.570158 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:05:17.570222 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:05:17.570445 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:05:17.570446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:05:17.570447 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:05:17.570499 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.570500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.571328 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.571328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.572589 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:05:17.572594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.573466 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.573467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:05:17.573552 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.573578 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 09:05:17.573652 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:05:17.573705 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":12,"thread_start_us":150,"threads_started":1}
I20260928 09:05:17.574227 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ee887149d47c5a688b9b7bf05656d" candidate_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4275d62117064a57b6afe3f1a6fa2801" is_pre_election: true
I20260928 09:05:17.574424 10907 raft_consensus.cc:2454] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3dfd95fbd94f5b90b89bfcd43e3d57 in term 0.
I20260928 09:05:17.574373 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ee887149d47c5a688b9b7bf05656d" candidate_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" is_pre_election: true
I20260928 09:05:17.574605 10773 raft_consensus.cc:2454] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3dfd95fbd94f5b90b89bfcd43e3d57 in term 0.
I20260928 09:05:17.574679 10976 leader_election.cc:302] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4275d62117064a57b6afe3f1a6fa2801, 6f3dfd95fbd94f5b90b89bfcd43e3d57; no voters:
I20260928 09:05:17.574818 11110 raft_consensus.cc:2781] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:05:17.574887 11110 raft_consensus.cc:482] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:05:17.574921 11110 raft_consensus.cc:3037] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:05:17.575670 11110 raft_consensus.cc:504] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.575820 11110 leader_election.cc:288] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [CANDIDATE]: Term 1 election: Requested vote from peers 4275d62117064a57b6afe3f1a6fa2801 (127.0.98.2:45389), 5dc191a6a92c4f5fa8cf335de13557aa (127.0.98.1:41603)
I20260928 09:05:17.576004 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ee887149d47c5a688b9b7bf05656d" candidate_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4275d62117064a57b6afe3f1a6fa2801"
I20260928 09:05:17.576004 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ee887149d47c5a688b9b7bf05656d" candidate_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc191a6a92c4f5fa8cf335de13557aa"
I20260928 09:05:17.576081 10907 raft_consensus.cc:3037] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:05:17.576102 10773 raft_consensus.cc:3037] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:05:17.576803 10907 raft_consensus.cc:2454] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3dfd95fbd94f5b90b89bfcd43e3d57 in term 1.
I20260928 09:05:17.576894 10773 raft_consensus.cc:2454] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3dfd95fbd94f5b90b89bfcd43e3d57 in term 1.
I20260928 09:05:17.576977 10976 leader_election.cc:302] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4275d62117064a57b6afe3f1a6fa2801, 6f3dfd95fbd94f5b90b89bfcd43e3d57; no voters:
I20260928 09:05:17.577076 11110 raft_consensus.cc:2781] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:05:17.577250 11110 raft_consensus.cc:686] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3dfd95fbd94f5b90b89bfcd43e3d57, State: Running, Role: LEADER
I20260928 09:05:17.577335 11110 consensus_queue.cc:237] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } }
I20260928 09:05:17.578317 10628 catalog_manager.cc:5495] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f3dfd95fbd94f5b90b89bfcd43e3d57 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3dfd95fbd94f5b90b89bfcd43e3d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20260928 09:05:17.591709 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.589268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.0.98.3:44965)
0928 09:05:17.589312 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:05:17.589314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:05:17.589324 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:05:17.589368 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:05:17.589369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:05:17.589386 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:05:17.589452 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:05:17.589618 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.590092 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.590893 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:05:17.591136 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:05:17.591240 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:05:17.591409 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:05:17.591530 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:05:17.591597 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:05:17.591630 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 09:05:17.591649 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:05:17.593719 10773 raft_consensus.cc:1261] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa [term 1 FOLLOWER]: Refusing update from remote peer 6f3dfd95fbd94f5b90b89bfcd43e3d57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:05:17.593739 10907 raft_consensus.cc:1261] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 1 FOLLOWER]: Refusing update from remote peer 6f3dfd95fbd94f5b90b89bfcd43e3d57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:05:17.594094 11110 consensus_queue.cc:1035] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4275d62117064a57b6afe3f1a6fa2801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:05:17.594244 11114 consensus_queue.cc:1035] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc191a6a92c4f5fa8cf335de13557aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 09:05:17.594533 10820 tablet.cc:2369] T d34ee887149d47c5a688b9b7bf05656d P 5dc191a6a92c4f5fa8cf335de13557aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:05:17.597535 11118 mvcc.cc:205] Tried to move back new op lower bound from 7334241556860985344 to 7334241556797521920. Current Snapshot: MvccSnapshot[applied={T|T < 7334241556860985344}]
I20260928 09:05:17.597671 11119 mvcc.cc:205] Tried to move back new op lower bound from 7334241556860985344 to 7334241556797521920. Current Snapshot: MvccSnapshot[applied={T|T < 7334241556860985344 or (T in {7334241556860985344})}]
I20260928 09:05:17.599025 10627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:05:17.599120 10627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:05:17.599862 10627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e97a1c571b6e4b41b9e977f0c35a4f4a: Sending DeleteTablet for 3 replicas of tablet d34ee887149d47c5a688b9b7bf05656d
I20260928 09:05:17.600152 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet d34ee887149d47c5a688b9b7bf05656d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53230
I20260928 09:05:17.600253 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet d34ee887149d47c5a688b9b7bf05656d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42792
I20260928 09:05:17.600265 10753 tablet_service.cc:1514] Processing DeleteTablet for tablet d34ee887149d47c5a688b9b7bf05656d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48646
I20260928 09:05:17.600482 11127 tablet_replica.cc:330] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: stopping tablet replica
I20260928 09:05:17.600612 11128 tablet_replica.cc:330] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: stopping tablet replica
I20260928 09:05:17.600704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10690
I20260928 09:05:17.600841 11127 raft_consensus.cc:2229] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:05:17.600740 11128 raft_consensus.cc:2229] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:05:17.601071 11127 raft_consensus.cc:2258] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:05:17.601229 11128 raft_consensus.cc:2258] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:05:17.601953 11127 ts_tablet_manager.cc:1905] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:05:17.603772 11128 ts_tablet_manager.cc:1905] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:05:17.603803 11127 ts_tablet_manager.cc:1918] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:05:17.603874 11127 log.cc:1196] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/d34ee887149d47c5a688b9b7bf05656d
I20260928 09:05:17.604154 11127 ts_tablet_manager.cc:1939] T d34ee887149d47c5a688b9b7bf05656d P 6f3dfd95fbd94f5b90b89bfcd43e3d57: Deleting consensus metadata
I20260928 09:05:17.605419 11128 ts_tablet_manager.cc:1918] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:05:17.605492 11128 log.cc:1196] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/d34ee887149d47c5a688b9b7bf05656d
I20260928 09:05:17.605558 10615 catalog_manager.cc:4841] TS 6f3dfd95fbd94f5b90b89bfcd43e3d57 (127.0.98.3:44965): tablet d34ee887149d47c5a688b9b7bf05656d (table test-table [id=4da022ae25e84fda9cbafd50026e0a60]) successfully deleted
I20260928 09:05:17.605775 11128 ts_tablet_manager.cc:1939] T d34ee887149d47c5a688b9b7bf05656d P 4275d62117064a57b6afe3f1a6fa2801: Deleting consensus metadata
I20260928 09:05:17.606268 10614 catalog_manager.cc:4841] TS 4275d62117064a57b6afe3f1a6fa2801 (127.0.98.2:45389): tablet d34ee887149d47c5a688b9b7bf05656d (table test-table [id=4da022ae25e84fda9cbafd50026e0a60]) successfully deleted
W20260928 09:05:17.607045 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41603 (error 108)
I20260928 09:05:17.607213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10824
I20260928 09:05:17.610085 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:05:17.609941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41603 (local address 127.0.0.1:48648)
0928 09:05:17.610001 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 09:05:17.610050 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 09:05:17.610271 10614 catalog_manager.cc:4584] TS 5dc191a6a92c4f5fa8cf335de13557aa (127.0.98.1:41603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d34ee887149d47c5a688b9b7bf05656d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41603: connect: Connection refused (error 111)
I20260928 09:05:17.613122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10958
I20260928 09:05:17.617894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 10596
2026-09-28T09:05:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2032 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[11133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[11133](info): set up 2 sockets
Sep 28 09:05:17 dist-test-slave-smns krb5kdc[11133](info): commencing operation
krb5kdc: starting...
W20260928 09:05:18.674690 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:05:18 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:05:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 09:05:18.840667 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAWHStC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/postgres -l logfile start
2026-09-28 09:05:21.549 UTC [11163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 09:05:21.549 UTC [11163] LOG: listening on IPv4 address "127.0.98.20", port 36605
2026-09-28 09:05:21.553 UTC [11163] LOG: listening on Unix socket "/tmp/.s.PGSQL.36605"
2026-09-28 09:05:21.575 UTC [11166] LOG: database system was shut down at 2026-09-28 09:05:19 UTC
2026-09-28 09:05:21.579 UTC [11163] LOG: database system is ready to accept connections
I20260928 09:05:22.534584 392 mini_postgres.cc:96] Postgres bound to 36605
2026-09-28 09:05:22.540 UTC [11174] FATAL: database "slave" does not exist
127.0.98.20:36605 - accepting connections
I20260928 09:05:22.541270 392 mini_ranger.cc:161] Starting Ranger...
I20260928 09:05:22.556838 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 09:05:23.111490 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 09:05:23.111591 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-admin
2026-09-28 09:05:23,396 [I] DB FLAVOR :POSTGRES
2026-09-28 09:05:23,397 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:05:23,397 [I] Checking connection..
2026-09-28 09:05:23,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 09:05:23,664 [I] Checking connection passed.
2026-09-28 09:05:23,664 [I] --------- Verifying version history table ---------
2026-09-28 09:05:23,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:05:23,932 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:05:23,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:05:24,171 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:05:24,171 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 09:05:24,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 09:05:24.437 UTC [11238] WARNING: there is no transaction in progress
2026-09-28 09:05:24,443 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 09:05:24,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:05:24,699 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 09:05:24,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:05:24,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 09:05:24,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 09:05:25,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-smns.c.gcp-upstream.internal','N') ;"
2026-09-28 09:05:25,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:05:25,676 [I] Table x_portal_user does not exist in database ranger
2026-09-28 09:05:25,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:05:25,911 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 09:05:25,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:05:26,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:05:26,388 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 09:05:26,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 09:05:27.391 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.408 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.425 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.441 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.460 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.479 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.541 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.549 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.565 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.576 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.586 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.601 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.611 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.617 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.626 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:27.633 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 09:05:28,335 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 09:05:28,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 09:05:28,561 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 09:05:28,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:05:28,819 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 09:05:28,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:05:29,061 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 09:05:29,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:05:29,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:05:29,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-smns.c.gcp-upstream.internal';"
2026-09-28 09:05:29,826 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 09:05:29,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 09:05:30,086 [I] DB_PATCHES have already been applied
I20260928 09:05:30.092419 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAWHStC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 09:05:30.092485 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 09:05:30.095472 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:43881
Sep 28, 2026 9:05:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:05:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:05:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:05:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:05:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:05:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:05:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 09:05:31.205196 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.664s user 0.000s sys 0.009s
Sep 28, 2026 9:05:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:05:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:05:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:05:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:05:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 09:05:52.665022 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 09:05:52 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586352, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 09:05:52.749848 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 09:05:52.758958 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 09:05:53.751901 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 09:05:53.751968 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms
I20260928 09:05:53.751981 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36605
2026-09-28 09:05:53,926 [I] DB FLAVOR :POSTGRES
2026-09-28 09:05:53,926 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:05:53,926 [I] Checking connection
2026-09-28 09:05:53,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 09:05:54,180 [I] connection success
2026-09-28 09:05:54,181 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 09:05:54,181 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 09:05:54,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 09:05:54,424 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 09:05:54,425 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:05:54,425 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 09:05:54,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36605/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 09:05:54,705 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 09:05:55.157454 392 mini_ranger_kms.cc:310] Created kms service
I20260928 09:05:55.336843 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 09:05:55.377828 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 09:05:55 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586352, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:05:55.768890 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 09:05:55.769003 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 09:05:55.769025 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 09:05:55.772992 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11526
I20260928 09:05:55.773138 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:41891
Sep 28, 2026 9:05:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:05:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:05:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:05:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:05:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:05:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 09:05:58.114161 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.364s user 0.011s sys 0.146s
I20260928 09:05:58.114274 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 09:05:58.114358 392 mini_ranger_kms.cc:401] 127.0.98.20:41891/kms/v1/keys
Sep 28 09:05:58 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586358, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 09:05:58 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586358, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:05:59 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586358, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:06:01 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:06:01.612226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46091
--ranger_config_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41891/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:01.695590 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:01.695799 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:01.695822 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:01.697005 11577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:01.697047 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:01.697062 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:01.697073 11577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:01.697086 11577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:01.698740 11577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:41891/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46091
--ranger_config_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:01.698988 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:01.699239 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:01.704247 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:01.704241 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:01.704217 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:01.704559 11577 server_base.cc:1034] running on GCE node
I20260928 09:06:01.704797 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:01.705114 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:01.706298 11577 hybrid_clock.cc:648] HybridClock initialized: now 1790586361706273 us; error 39 us; skew 500 ppm
Sep 28 09:06:01 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:01.708771 11577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:06:01.709167 11577 webserver.cc:460] Webserver started at http://127.0.98.62:34489/ using document root <none> and password file <none>
I20260928 09:06:01.709350 11577 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:01.709385 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:01.709474 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:01 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:01.766742 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "3ffe5794da854dcca17a56b16fe96b3f"
format_stamp: "Formatted at 2026-09-28 09:06:01 on dist-test-slave-smns"
server_key: "90e3849f95cc20c58772cd80332a2c75"
server_key_iv: "10029de1dea72bd2b2be645c9517687d"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:01.767191 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ffe5794da854dcca17a56b16fe96b3f"
format_stamp: "Formatted at 2026-09-28 09:06:01 on dist-test-slave-smns"
server_key: "90e3849f95cc20c58772cd80332a2c75"
server_key_iv: "10029de1dea72bd2b2be645c9517687d"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:01.768725 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:06:01.783846 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:01.784045 11577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:06:01.784132 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "3ffe5794da854dcca17a56b16fe96b3f"
format_stamp: "Formatted at 2026-09-28 09:06:01 on dist-test-slave-smns"
server_key: "90e3849f95cc20c58772cd80332a2c75"
server_key_iv: "10029de1dea72bd2b2be645c9517687d"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:01.784233 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:01.796620 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:01.798727 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:01.798874 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:01.804104 11577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46091
I20260928 09:06:01.804131 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46091 every 8 connection(s)
I20260928 09:06:01.804487 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:06:01.805176 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:01.807548 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f: Bootstrap starting.
I20260928 09:06:01.808036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 11577
I20260928 09:06:01.808140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:06:01.808211 11645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:01.808329 392 external_mini_cluster.cc:1428] Setting key bac9aeb5bfe60aefad58e7aa1900065f
I20260928 09:06:01.808573 11645 log.cc:824] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:01.809429 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f: No bootstrap required, opened a new log
I20260928 09:06:01.811275 11645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe5794da854dcca17a56b16fe96b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260928 09:06:01.811439 11645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:01.811501 11645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffe5794da854dcca17a56b16fe96b3f, State: Initialized, Role: FOLLOWER
I20260928 09:06:01.811676 11645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe5794da854dcca17a56b16fe96b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260928 09:06:01.811772 11645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:06:01.811826 11645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:06:01.811898 11645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:01.812438 11645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe5794da854dcca17a56b16fe96b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260928 09:06:01.812563 11645 leader_election.cc:302] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ffe5794da854dcca17a56b16fe96b3f; no voters:
I20260928 09:06:01.812780 11645 leader_election.cc:288] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:06:01.812801 11650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:01.813041 11650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [term 1 LEADER]: Becoming Leader. State: Replica: 3ffe5794da854dcca17a56b16fe96b3f, State: Running, Role: LEADER
I20260928 09:06:01.813090 11645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:01.813199 11650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe5794da854dcca17a56b16fe96b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260928 09:06:01.813611 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ffe5794da854dcca17a56b16fe96b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe5794da854dcca17a56b16fe96b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } } }
I20260928 09:06:01.813767 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:01.813942 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ffe5794da854dcca17a56b16fe96b3f. Latest consensus state: current_term: 1 leader_uuid: "3ffe5794da854dcca17a56b16fe96b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe5794da854dcca17a56b16fe96b3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } } }
I20260928 09:06:01.814054 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:01.814577 11645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 09:06:01 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:01.817798 11659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:06:01.819093 11659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:06:01.822374 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:01.808944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.0.98.62:46091)
0928 09:06:01.809102 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:01.809113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:01.809144 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:06:01.809759 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:01.809769 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:01.809838 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:01.809993 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:01.810809 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:01.812587 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:01.813751 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:01.814381 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:01.814482 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:01.817883 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:01.817922 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:01.817922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:01.817960 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:01.820659 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:01.821055 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:01.821058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:01.821059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:01.821151 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:01.821367 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:01.821367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:01.821368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:01.821737 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:01.821892 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:01.822007 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 09:06:01.822164 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 09:06:01.827765 11659 catalog_manager.cc:1347] Generated new cluster ID: a019adbd3d3c468ba1018dde1324d176
I20260928 09:06:01.827826 11659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:06:01.843868 11659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:06:01.844692 11659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:06:01.849097 11659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f: Generated new TSK 0
I20260928 09:06:01.849296 11659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:06:02.513734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:02.634245 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:02.634474 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:02.634510 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:02.635660 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:02.635727 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:02.635775 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:06:02.637359 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:41891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:02.637676 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:02.637909 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:02.643042 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:02.643042 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:02.643160 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:02.643508 11690 server_base.cc:1034] running on GCE node
I20260928 09:06:02.643723 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:02.643975 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:02.645149 11690 hybrid_clock.cc:648] HybridClock initialized: now 1790586362645134 us; error 34 us; skew 500 ppm
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:02.648113 11690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:06:02.648711 11690 webserver.cc:460] Webserver started at http://127.0.98.1:32777/ using document root <none> and password file <none>
I20260928 09:06:02.648999 11690 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:02.649068 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:02.649199 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:02.665172 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "edd9ab0781fd4b74b17662a106907fb4"
format_stamp: "Formatted at 2026-09-28 09:06:02 on dist-test-slave-smns"
server_key: "92f60f6077da9372a640c9bb4b426cf2"
server_key_iv: "25480dcfc14a9f9a5fc61f53ea93201d"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:02.665668 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edd9ab0781fd4b74b17662a106907fb4"
format_stamp: "Formatted at 2026-09-28 09:06:02 on dist-test-slave-smns"
server_key: "92f60f6077da9372a640c9bb4b426cf2"
server_key_iv: "25480dcfc14a9f9a5fc61f53ea93201d"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:02.667646 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 09:06:02.689698 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:02.689949 11690 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260928 09:06:02.690514 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "edd9ab0781fd4b74b17662a106907fb4"
format_stamp: "Formatted at 2026-09-28 09:06:02 on dist-test-slave-smns"
server_key: "92f60f6077da9372a640c9bb4b426cf2"
server_key_iv: "25480dcfc14a9f9a5fc61f53ea93201d"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:02.690778 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:02.704500 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:02.707008 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:02.707166 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:02.707407 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:02.707805 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:02.707862 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:02.707894 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:02.707912 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:02.715715 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34119
I20260928 09:06:02.715793 11820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34119 every 8 connection(s)
I20260928 09:06:02.716238 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:02.723691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 11690
I20260928 09:06:02.723800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:06:02.724026 392 external_mini_cluster.cc:1428] Setting key b8dc254a5df0b9588c6ae391616846d8
I20260928 09:06:02.727164 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:02.717086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58621 (local address 127.0.98.62:46091)
0928 09:06:02.717228 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:02.717230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:02.717826 (+ 596us) server_negotiation.cc:407] Connection header received
0928 09:06:02.718460 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:02.718461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:02.718478 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:02.718579 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:02.719291 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.720080 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.721055 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.721374 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.721456 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:02.723598 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:02.723600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:02.723600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:02.723608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:02.725470 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:02.725986 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:02.725987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:02.725987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:02.726052 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:02.726357 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:02.726358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:02.726359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:02.726494 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:02.726566 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:02.726972 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 09:06:02.727062 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260928 09:06:02.727842 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:02.717061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46091 (local address 127.0.98.1:58621)
0928 09:06:02.717666 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:02.717714 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:02.718334 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:02.718666 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:02.718670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:02.718905 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:02.719176 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:02.719183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.720175 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.720176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:02.720963 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:02.720965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.721465 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.721465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:02.721647 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:02.722227 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:02.722235 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:02.723465 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:02.725606 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:02.725613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:02.725615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:02.725897 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:02.726133 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:02.726134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:02.726134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:02.726275 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:02.726604 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:02.726610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:02.726796 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 09:06:02.727411 (+ 615us) client_negotiation.cc:240] Negotiation successful
0928 09:06:02.727598 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":525,"threads_started":1}
I20260928 09:06:02.728363 11821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46091
I20260928 09:06:02.728480 11821 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:02.728696 11821 heartbeater.cc:511] Master 127.0.98.62:46091 requested a full tablet report, sending...
I20260928 09:06:02.729333 11609 ts_manager.cc:194] Registered new tserver with Master: edd9ab0781fd4b74b17662a106907fb4 (127.0.98.1:34119)
I20260928 09:06:02.730340 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58621
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:06:02.757958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 09:06:02.878518 11827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:02.878872 11827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:02.878974 11827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:02.880906 11827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:02.881022 11827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:02.881106 11827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:06:02.883478 11827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:41891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:02.884013 11827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:02.884389 11827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:02.891904 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:02.892294 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:02.892560 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:02.893504 11827 server_base.cc:1034] running on GCE node
I20260928 09:06:02.893749 11827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:02.894047 11827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:02.901446 11827 hybrid_clock.cc:648] HybridClock initialized: now 1790586362901394 us; error 59 us; skew 500 ppm
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:02.904891 11827 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:06:02.905474 11827 webserver.cc:460] Webserver started at http://127.0.98.2:46687/ using document root <none> and password file <none>
I20260928 09:06:02.905809 11827 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:02.905889 11827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:02.906085 11827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:02.928591 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "32873b7abdf547eb9ba621e842fdeb5c"
format_stamp: "Formatted at 2026-09-28 09:06:02 on dist-test-slave-smns"
server_key: "eb2fece33059dab3537cb7d010f8ab4a"
server_key_iv: "d91c30ff7798acdef0d4bad18ca60430"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:02.929080 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32873b7abdf547eb9ba621e842fdeb5c"
format_stamp: "Formatted at 2026-09-28 09:06:02 on dist-test-slave-smns"
server_key: "eb2fece33059dab3537cb7d010f8ab4a"
server_key_iv: "d91c30ff7798acdef0d4bad18ca60430"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:02.931476 11827 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 09:06:02.948352 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:02.948547 11827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 09:06:02.948637 11827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "32873b7abdf547eb9ba621e842fdeb5c"
format_stamp: "Formatted at 2026-09-28 09:06:02 on dist-test-slave-smns"
server_key: "eb2fece33059dab3537cb7d010f8ab4a"
server_key_iv: "d91c30ff7798acdef0d4bad18ca60430"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:02.948745 11827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:02.964416 11827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:02.967142 11827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:02.967358 11827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:02.967641 11827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:02.968094 11827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:02.968185 11827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:02.968261 11827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:02.968322 11827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:02.976611 11827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43745
I20260928 09:06:02.977171 11827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:06:02.979374 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43745 every 8 connection(s)
I20260928 09:06:02.985605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 11827
I20260928 09:06:02.985715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:06:02.985944 392 external_mini_cluster.cc:1428] Setting key c105c6c91a73f09979569dfa3ad28160
Sep 28 09:06:02 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:06:02.998728 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:02.980501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47197 (local address 127.0.98.62:46091)
0928 09:06:02.981025 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:02.981028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:02.985830 (+ 4802us) server_negotiation.cc:407] Connection header received
0928 09:06:02.987068 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:02.987070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:02.987082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:02.987114 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:02.988015 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.988835 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.990726 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.991027 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.991078 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:02.994911 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:02.994913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:02.994913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:02.994922 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:02.996930 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:02.997502 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:02.997504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:02.997504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:02.997584 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:02.997892 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:02.997893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:02.997893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:02.998052 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:02.998110 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:02.998525 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 09:06:02.998625 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
I20260928 09:06:02.999348 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:02.980200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46091 (local address 127.0.98.2:47197)
0928 09:06:02.985644 (+ 5444us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:02.985711 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:02.986950 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:02.987313 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:02.987317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:02.987592 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:02.987907 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:02.987914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.988972 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.988973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:02.990608 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:02.990611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:02.991176 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:02.991177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:02.991333 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:02.992760 (+ 1427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:02.992771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:02.994749 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:02.997047 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:02.997054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:02.997057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:02.997401 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:02.997656 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:02.997657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:02.997658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:02.997805 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:02.998150 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:02.998156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:02.998350 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 09:06:02.998888 (+ 538us) client_negotiation.cc:240] Negotiation successful
0928 09:06:02.999089 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5385,"thread_start_us":95,"threads_started":1}
I20260928 09:06:02.999964 11959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46091
I20260928 09:06:03.000087 11959 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:03.000303 11959 heartbeater.cc:511] Master 127.0.98.62:46091 requested a full tablet report, sending...
I20260928 09:06:03.000772 11609 ts_manager.cc:194] Registered new tserver with Master: 32873b7abdf547eb9ba621e842fdeb5c (127.0.98.2:43745)
I20260928 09:06:03.001289 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:06:03.021616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:03.148489 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:03.148928 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:03.149055 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:03.151346 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:03.151481 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:03.151566 11964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:06:03.154040 11964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:41891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:03.154793 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:03.155144 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:03.161937 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:03.162532 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:03.163230 11964 server_base.cc:1034] running on GCE node
W20260928 09:06:03.166301 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:03.166635 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:03.167002 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:03.168183 11964 hybrid_clock.cc:648] HybridClock initialized: now 1790586363168157 us; error 41 us; skew 500 ppm
Sep 28 09:06:03 dist-test-slave-smns krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:03.172668 11964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:06:03.173280 11964 webserver.cc:460] Webserver started at http://127.0.98.3:37107/ using document root <none> and password file <none>
I20260928 09:06:03.173653 11964 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:03.173725 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:03.173868 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:03 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:03.188633 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "0bfafa4749804e4ca4b340b484c71113"
format_stamp: "Formatted at 2026-09-28 09:06:03 on dist-test-slave-smns"
server_key: "623a307110adb06ae4fd13ac09d34944"
server_key_iv: "5ef7877334569be598728a02ac2551eb"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:03.189131 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0bfafa4749804e4ca4b340b484c71113"
format_stamp: "Formatted at 2026-09-28 09:06:03 on dist-test-slave-smns"
server_key: "623a307110adb06ae4fd13ac09d34944"
server_key_iv: "5ef7877334569be598728a02ac2551eb"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:03.191205 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 09:06:03.205119 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:03.205353 11964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:06:03.205448 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "0bfafa4749804e4ca4b340b484c71113"
format_stamp: "Formatted at 2026-09-28 09:06:03 on dist-test-slave-smns"
server_key: "623a307110adb06ae4fd13ac09d34944"
server_key_iv: "5ef7877334569be598728a02ac2551eb"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:03.205564 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:03.242708 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:03.246482 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:03.246639 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:03.246892 11964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:03.247254 11964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:03.247298 11964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:03.247330 11964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:03.247347 11964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:03.254841 11964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40163
I20260928 09:06:03.255263 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:06:03.259051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 11964
I20260928 09:06:03.259181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:06:03.259382 392 external_mini_cluster.cc:1428] Setting key 48101a5b3a879a40ced7398623f9636e
I20260928 09:06:03.265743 12092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40163 every 8 connection(s)
Sep 28 09:06:03 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:03.282908 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.266501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48659 (local address 127.0.98.62:46091)
0928 09:06:03.266551 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.266554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.269858 (+ 3304us) server_negotiation.cc:407] Connection header received
0928 09:06:03.270562 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.270563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.270576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.270606 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:03.272104 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.272835 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.274339 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.274621 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.274664 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.278839 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:03.278841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:03.278842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:03.278851 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:03.280615 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:03.281144 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:03.281145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:03.281146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:03.281202 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:03.282097 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:03.282098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:03.282099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:03.282251 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:03.282301 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.282700 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.282794 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:06:03.283731 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.266526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46091 (local address 127.0.98.3:48659)
0928 09:06:03.269695 (+ 3169us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.269755 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:03.270436 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:03.271367 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:03.271372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:03.271675 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:03.271984 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.271992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.272967 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.272968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.274232 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.274235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.274758 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.274759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.274896 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.275527 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:03.275539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:03.277203 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:03.280726 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:03.280736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:03.280739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:03.281057 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:03.281266 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:03.281266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:03.281267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:03.282023 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:03.282333 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:03.282339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:03.282546 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 09:06:03.283293 (+ 747us) client_negotiation.cc:240] Negotiation successful
0928 09:06:03.283483 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3111,"thread_start_us":106,"threads_started":1}
I20260928 09:06:03.284368 12093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46091
I20260928 09:06:03.284472 12093 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:03.284663 12093 heartbeater.cc:511] Master 127.0.98.62:46091 requested a full tablet report, sending...
I20260928 09:06:03.285064 11609 ts_manager.cc:194] Registered new tserver with Master: 0bfafa4749804e4ca4b340b484c71113 (127.0.98.3:40163)
I20260928 09:06:03.285630 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48659
I20260928 09:06:03.292719 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:06:03.307189 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.297046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.62:46091)
0928 09:06:03.297099 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.297101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.297447 (+ 346us) server_negotiation.cc:407] Connection header received
0928 09:06:03.297598 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.297600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.297615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.297648 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:03.298465 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.299230 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.300672 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.300961 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.301010 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.302750 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:03.302752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:03.302752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:03.302760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:03.305794 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:03.306231 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:03.306232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:03.306233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:03.306290 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:03.306508 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:03.306508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:03.306509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:03.306641 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:03.306716 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.306993 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.307081 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:06:03.309665 11609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:06:03.312160 11609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:06:03.323962 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.317307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40163 (local address 127.0.0.1:51584)
0928 09:06:03.318000 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.318043 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:03.318112 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:03.318553 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:03.318557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:03.318559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:03.318628 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.318630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.319940 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.319941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.322564 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.322566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.323641 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.323646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.323727 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.323742 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 09:06:03.323821 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 09:06:03.323875 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1093,"thread_start_us":98,"threads_started":1}
I20260928 09:06:03.324582 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.317418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.0.98.3:40163)
0928 09:06:03.318295 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.318301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.318310 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:06:03.318349 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.318351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.318367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.318454 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:03.318713 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.319831 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.322687 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.323422 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.323497 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.323577 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.324456 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.324501 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":82,"threads_started":1}
I20260928 09:06:03.326278 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.317839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43745 (local address 127.0.0.1:50164)
0928 09:06:03.318252 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.318260 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:03.318291 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:03.318623 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:03.318624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:03.318626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:03.318676 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.318677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.320252 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.320253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.321351 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.321354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.326051 (+ 4697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.326052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.326115 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.326119 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:06:03.326165 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 09:06:03.326205 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":62,"threads_started":1}
I20260928 09:06:03.327299 12027 tablet_service.cc:1467] Processing CreateTablet for tablet dc36922ab8f54c319a942b7554ad3589 (DEFAULT_TABLE table=test-table [id=97d9bbc8313f48b8a5f0969a17d332ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:06:03.327682 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc36922ab8f54c319a942b7554ad3589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:03.332490 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.317963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.0.98.2:43745)
0928 09:06:03.318408 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.318413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.318423 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:06:03.318465 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.318467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.318481 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.318550 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:03.318742 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.320170 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.325104 (+ 4934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.325870 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.325944 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.326015 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.332318 (+ 6303us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.332396 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":71,"threads_started":1}
I20260928 09:06:03.334847 11893 tablet_service.cc:1467] Processing CreateTablet for tablet dc36922ab8f54c319a942b7554ad3589 (DEFAULT_TABLE table=test-table [id=97d9bbc8313f48b8a5f0969a17d332ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:06:03.335206 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc36922ab8f54c319a942b7554ad3589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:03.336993 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.317651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34119 (local address 127.0.0.1:58552)
0928 09:06:03.318101 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.318111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:03.318159 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:03.332793 (+ 14634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:03.332795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:03.332798 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:03.332868 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.332870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.334272 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.334273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.335343 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.335346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.336749 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.336750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.336827 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.336834 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 09:06:03.336865 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 09:06:03.336911 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":66,"threads_started":1}
I20260928 09:06:03.337410 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.317735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.0.98.1:34119)
0928 09:06:03.332103 (+ 14368us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.332109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.332123 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:06:03.332163 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.332165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.332184 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.332952 (+ 768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:03.333021 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.334096 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.335463 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.336641 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.337170 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.337238 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.337265 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.337304 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14309,"thread_start_us":46,"threads_started":1}
I20260928 09:06:03.338050 12112 tablet_bootstrap.cc:492] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Bootstrap starting.
I20260928 09:06:03.338081 11755 tablet_service.cc:1467] Processing CreateTablet for tablet dc36922ab8f54c319a942b7554ad3589 (DEFAULT_TABLE table=test-table [id=97d9bbc8313f48b8a5f0969a17d332ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:06:03.338406 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc36922ab8f54c319a942b7554ad3589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:03.339171 12112 tablet_bootstrap.cc:654] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:03.339488 12112 log.cc:824] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:03.346961 12114 tablet_bootstrap.cc:492] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Bootstrap starting.
I20260928 09:06:03.347507 12112 tablet_bootstrap.cc:492] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: No bootstrap required, opened a new log
I20260928 09:06:03.347865 12112 ts_tablet_manager.cc:1397] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 09:06:03.348753 12115 tablet_bootstrap.cc:492] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4: Bootstrap starting.
I20260928 09:06:03.350306 12114 tablet_bootstrap.cc:654] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:03.349532 12115 tablet_bootstrap.cc:654] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:03.350464 12112 raft_consensus.cc:348] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.350656 12112 raft_consensus.cc:374] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:03.350698 12112 raft_consensus.cc:729] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bfafa4749804e4ca4b340b484c71113, State: Initialized, Role: FOLLOWER
I20260928 09:06:03.350860 12112 consensus_queue.cc:260] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.351138 12112 ts_tablet_manager.cc:1428] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:06:03.351270 12093 heartbeater.cc:503] Master 127.0.98.62:46091 was elected leader, sending a full tablet report...
I20260928 09:06:03.351393 12114 log.cc:824] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:03.351498 12115 log.cc:824] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:03.352799 12115 tablet_bootstrap.cc:492] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4: No bootstrap required, opened a new log
I20260928 09:06:03.352880 12115 ts_tablet_manager.cc:1397] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 09:06:03.355818 12115 raft_consensus.cc:348] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.356004 12115 raft_consensus.cc:374] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:03.356045 12115 raft_consensus.cc:729] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd9ab0781fd4b74b17662a106907fb4, State: Initialized, Role: FOLLOWER
I20260928 09:06:03.356189 12115 consensus_queue.cc:260] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.357303 12115 ts_tablet_manager.cc:1428] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 09:06:03.357826 12114 tablet_bootstrap.cc:492] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: No bootstrap required, opened a new log
I20260928 09:06:03.357962 12114 ts_tablet_manager.cc:1397] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260928 09:06:03.360141 12114 raft_consensus.cc:348] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.360396 12114 raft_consensus.cc:374] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:03.360502 12114 raft_consensus.cc:729] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32873b7abdf547eb9ba621e842fdeb5c, State: Initialized, Role: FOLLOWER
I20260928 09:06:03.360703 12114 consensus_queue.cc:260] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.361050 12114 ts_tablet_manager.cc:1428] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:06:03.361207 11959 heartbeater.cc:503] Master 127.0.98.62:46091 was elected leader, sending a full tablet report...
I20260928 09:06:03.361979 11821 heartbeater.cc:503] Master 127.0.98.62:46091 was elected leader, sending a full tablet report...
I20260928 09:06:03.397697 12119 raft_consensus.cc:482] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:06:03.397921 12119 raft_consensus.cc:504] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.398898 12119 leader_election.cc:288] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bfafa4749804e4ca4b340b484c71113 (127.0.98.3:40163), 32873b7abdf547eb9ba621e842fdeb5c (127.0.98.2:43745)
I20260928 09:06:03.414268 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.398621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41195 (local address 127.0.98.3:40163)
0928 09:06:03.398666 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.398669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.405747 (+ 7078us) server_negotiation.cc:407] Connection header received
0928 09:06:03.405971 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.405973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.405985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.406014 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:03.407497 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.408231 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.413274 (+ 5043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.413866 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.413940 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.413952 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.414146 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.414189 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:06:03.414588 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.398550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40163 (local address 127.0.98.1:41195)
0928 09:06:03.405645 (+ 7095us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.405662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:03.405889 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:03.406081 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:03.406082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:03.406083 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:03.406175 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.406177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.409723 (+ 3546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.409724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.410759 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.410763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.413989 (+ 3226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.413991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.414066 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.414071 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:06:03.414422 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 09:06:03.414476 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7067,"thread_start_us":101,"threads_started":1}
I20260928 09:06:03.414934 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc36922ab8f54c319a942b7554ad3589" candidate_uuid: "edd9ab0781fd4b74b17662a106907fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bfafa4749804e4ca4b340b484c71113" is_pre_election: true
I20260928 09:06:03.415120 12038 raft_consensus.cc:2454] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edd9ab0781fd4b74b17662a106907fb4 in term 0.
I20260928 09:06:03.415309 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.398864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55083 (local address 127.0.98.2:43745)
0928 09:06:03.398907 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.398909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.405856 (+ 6947us) server_negotiation.cc:407] Connection header received
0928 09:06:03.406314 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.406315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.406324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.406352 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:03.406532 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.407353 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.413321 (+ 5968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.413971 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.414027 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.414038 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.415182 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.415237 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:06:03.415372 11708 leader_election.cc:302] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfafa4749804e4ca4b340b484c71113, edd9ab0781fd4b74b17662a106907fb4; no voters:
I20260928 09:06:03.415514 12119 raft_consensus.cc:2781] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:06:03.415556 12119 raft_consensus.cc:482] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:06:03.415577 12119 raft_consensus.cc:3037] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:03.416540 12119 raft_consensus.cc:504] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.416695 12119 leader_election.cc:288] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [CANDIDATE]: Term 1 election: Requested vote from peers 0bfafa4749804e4ca4b340b484c71113 (127.0.98.3:40163), 32873b7abdf547eb9ba621e842fdeb5c (127.0.98.2:43745)
I20260928 09:06:03.416909 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.398805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43745 (local address 127.0.98.1:55083)
0928 09:06:03.405804 (+ 6999us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.405812 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:03.406237 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:03.406409 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:03.406409 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:03.406410 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:03.406469 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.406470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.408361 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.408363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.409610 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:03.409614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.414768 (+ 5154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.414771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:03.414849 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.414853 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:06:03.416708 (+ 1855us) client_negotiation.cc:240] Negotiation successful
0928 09:06:03.416801 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6973,"thread_start_us":67,"threads_started":1}
I20260928 09:06:03.416932 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc36922ab8f54c319a942b7554ad3589" candidate_uuid: "edd9ab0781fd4b74b17662a106907fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bfafa4749804e4ca4b340b484c71113"
I20260928 09:06:03.417014 12038 raft_consensus.cc:3037] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:03.417901 11897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc36922ab8f54c319a942b7554ad3589" candidate_uuid: "edd9ab0781fd4b74b17662a106907fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32873b7abdf547eb9ba621e842fdeb5c" is_pre_election: true
I20260928 09:06:03.418061 11897 raft_consensus.cc:2454] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edd9ab0781fd4b74b17662a106907fb4 in term 0.
I20260928 09:06:03.418165 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc36922ab8f54c319a942b7554ad3589" candidate_uuid: "edd9ab0781fd4b74b17662a106907fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32873b7abdf547eb9ba621e842fdeb5c"
I20260928 09:06:03.418211 11910 raft_consensus.cc:3037] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:03.418212 12038 raft_consensus.cc:2454] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd9ab0781fd4b74b17662a106907fb4 in term 1.
I20260928 09:06:03.418399 11708 leader_election.cc:302] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfafa4749804e4ca4b340b484c71113, edd9ab0781fd4b74b17662a106907fb4; no voters:
I20260928 09:06:03.418671 12119 raft_consensus.cc:2781] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:03.418848 12119 raft_consensus.cc:686] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [term 1 LEADER]: Becoming Leader. State: Replica: edd9ab0781fd4b74b17662a106907fb4, State: Running, Role: LEADER
I20260928 09:06:03.418953 12119 consensus_queue.cc:237] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } }
I20260928 09:06:03.419117 11910 raft_consensus.cc:2454] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd9ab0781fd4b74b17662a106907fb4 in term 1.
I20260928 09:06:03.419885 11609 catalog_manager.cc:5495] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to edd9ab0781fd4b74b17662a106907fb4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "edd9ab0781fd4b74b17662a106907fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edd9ab0781fd4b74b17662a106907fb4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34119 } health_report { overall_health: HEALTHY } } }
Sep 28 09:06:03 dist-test-slave-smns krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:03.442821 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.439419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.0.98.1:34119)
0928 09:06:03.439459 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:03.439461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:03.439467 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:06:03.439507 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:03.439507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:03.439531 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:03.439596 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:06:03.439786 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.440546 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.441715 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:03.442013 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:03.442079 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:03.442283 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:06:03.442418 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:06:03.442488 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:03.442699 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 09:06:03.442722 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:06:03.449884 12038 raft_consensus.cc:1261] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 1 FOLLOWER]: Refusing update from remote peer edd9ab0781fd4b74b17662a106907fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:06:03.449883 11910 raft_consensus.cc:1261] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 1 FOLLOWER]: Refusing update from remote peer edd9ab0781fd4b74b17662a106907fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:06:03.450258 12123 consensus_queue.cc:1035] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfafa4749804e4ca4b340b484c71113" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:06:03.450572 12123 consensus_queue.cc:1035] T dc36922ab8f54c319a942b7554ad3589 P edd9ab0781fd4b74b17662a106907fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32873b7abdf547eb9ba621e842fdeb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:06:03.456892 12126 mvcc.cc:205] Tried to move back new op lower bound from 7334241744687575040 to 7334241744564641792. Current Snapshot: MvccSnapshot[applied={T|T < 7334241744687575040}]
I20260928 09:06:03.457453 12127 mvcc.cc:205] Tried to move back new op lower bound from 7334241744687575040 to 7334241744564641792. Current Snapshot: MvccSnapshot[applied={T|T < 7334241744687575040}]
I20260928 09:06:03.473076 11609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:06:03.473194 11609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:06:03.473423 12129 mvcc.cc:205] Tried to move back new op lower bound from 7334241744687575040 to 7334241744564641792. Current Snapshot: MvccSnapshot[applied={T|T < 7334241744687575040}]
I20260928 09:06:03.474261 11609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ffe5794da854dcca17a56b16fe96b3f: Sending DeleteTablet for 3 replicas of tablet dc36922ab8f54c319a942b7554ad3589
I20260928 09:06:03.474694 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet dc36922ab8f54c319a942b7554ad3589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:06:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51584
I20260928 09:06:03.474968 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet dc36922ab8f54c319a942b7554ad3589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:06:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58552
I20260928 09:06:03.475200 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet dc36922ab8f54c319a942b7554ad3589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:06:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50164
I20260928 09:06:03.476110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 11690
I20260928 09:06:03.485683 12135 tablet_replica.cc:330] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: stopping tablet replica
I20260928 09:06:03.485926 12137 tablet_replica.cc:330] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: stopping tablet replica
I20260928 09:06:03.486210 12137 raft_consensus.cc:2229] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:06:03.486483 12135 raft_consensus.cc:2229] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:06:03.486663 12137 raft_consensus.cc:2258] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:06:03.486770 12135 raft_consensus.cc:2258] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:06:03.487470 12137 ts_tablet_manager.cc:1905] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:06:03.489974 12135 ts_tablet_manager.cc:1905] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:06:03.490429 12137 ts_tablet_manager.cc:1918] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:06:03.490578 12137 log.cc:1196] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/dc36922ab8f54c319a942b7554ad3589
I20260928 09:06:03.490921 12137 ts_tablet_manager.cc:1939] T dc36922ab8f54c319a942b7554ad3589 P 32873b7abdf547eb9ba621e842fdeb5c: Deleting consensus metadata
I20260928 09:06:03.493084 12135 ts_tablet_manager.cc:1918] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:06:03.493173 12135 log.cc:1196] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/dc36922ab8f54c319a942b7554ad3589
I20260928 09:06:03.493784 11595 catalog_manager.cc:4841] TS 32873b7abdf547eb9ba621e842fdeb5c (127.0.98.2:43745): tablet dc36922ab8f54c319a942b7554ad3589 (table test-table [id=97d9bbc8313f48b8a5f0969a17d332ba]) successfully deleted
I20260928 09:06:03.494228 12135 ts_tablet_manager.cc:1939] T dc36922ab8f54c319a942b7554ad3589 P 0bfafa4749804e4ca4b340b484c71113: Deleting consensus metadata
I20260928 09:06:03.494711 11593 catalog_manager.cc:4841] TS 0bfafa4749804e4ca4b340b484c71113 (127.0.98.3:40163): tablet dc36922ab8f54c319a942b7554ad3589 (table test-table [id=97d9bbc8313f48b8a5f0969a17d332ba]) successfully deleted
W20260928 09:06:03.502032 11593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34119 (error 108)
I20260928 09:06:03.502916 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:03.502799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34119 (local address 127.0.0.1:58580)
0928 09:06:03.502847 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:03.502883 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 09:06:03.503103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 11827
W20260928 09:06:03.503180 11593 catalog_manager.cc:4584] TS edd9ab0781fd4b74b17662a106907fb4 (127.0.98.1:34119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc36922ab8f54c319a942b7554ad3589: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34119: connect: Connection refused (error 111)
I20260928 09:06:03.510368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 11964
I20260928 09:06:03.515337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 11577
I20260928 09:06:03.519851 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 09:06:03.864543 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 09:06:03.864648 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 09:06:04.209158 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 09:06:04.209 UTC [11163] LOG: received smart shutdown request
2026-09-28 09:06:04.212 UTC [11163] LOG: background worker "logical replication launcher" (PID 11172) exited with exit code 1
2026-09-28 09:06:04.213 UTC [11167] LOG: shutting down
2026-09-28 09:06:04.406 UTC [11163] LOG: database system is shut down
2026-09-28T09:06:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46896 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:06:04 dist-test-slave-smns krb5kdc[12147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:06:04 dist-test-slave-smns krb5kdc[12147](info): set up 2 sockets
Sep 28 09:06:04 dist-test-slave-smns krb5kdc[12147](info): commencing operation
krb5kdc: starting...
W20260928 09:06:05.557727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:06:05 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:06:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 09:06:05.733227 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAWHStC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/postgres -l logfile start
2026-09-28 09:06:08.400 UTC [12177] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 09:06:08.400 UTC [12177] LOG: listening on IPv4 address "127.0.98.20", port 50485
2026-09-28 09:06:08.404 UTC [12177] LOG: listening on Unix socket "/tmp/.s.PGSQL.50485"
2026-09-28 09:06:08.425 UTC [12180] LOG: database system was shut down at 2026-09-28 09:06:06 UTC
2026-09-28 09:06:08.428 UTC [12177] LOG: database system is ready to accept connections
I20260928 09:06:09.385461 392 mini_postgres.cc:96] Postgres bound to 50485
2026-09-28 09:06:09.391 UTC [12188] FATAL: database "slave" does not exist
127.0.98.20:50485 - accepting connections
I20260928 09:06:09.391974 392 mini_ranger.cc:161] Starting Ranger...
I20260928 09:06:09.406716 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 09:06:09.934130 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 09:06:09.934214 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-admin
2026-09-28 09:06:10,212 [I] DB FLAVOR :POSTGRES
2026-09-28 09:06:10,213 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:06:10,213 [I] Checking connection..
2026-09-28 09:06:10,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 09:06:10,483 [I] Checking connection passed.
2026-09-28 09:06:10,483 [I] --------- Verifying version history table ---------
2026-09-28 09:06:10,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:06:10,750 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:06:10,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:06:11,006 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:06:11,006 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 09:06:11,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 09:06:11.238 UTC [12252] WARNING: there is no transaction in progress
2026-09-28 09:06:11,244 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 09:06:11,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:06:11,510 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 09:06:11,510 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:06:11,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 09:06:11,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 09:06:11,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-smns.c.gcp-upstream.internal','N') ;"
2026-09-28 09:06:12,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:06:12,477 [I] Table x_portal_user does not exist in database ranger
2026-09-28 09:06:12,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:06:12,746 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 09:06:12,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:06:13,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:06:13,264 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 09:06:13,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 09:06:14.332 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.350 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.367 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.386 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.407 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.429 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.497 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.504 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.520 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.531 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.542 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.557 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.567 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.575 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.582 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:14.590 UTC [12360] WARNING: there is no transaction in progress
2026-09-28 09:06:15,350 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 09:06:15,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 09:06:15,613 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 09:06:15,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:06:15,879 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 09:06:15,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:06:16,123 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 09:06:16,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:06:16,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:06:16,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-smns.c.gcp-upstream.internal';"
2026-09-28 09:06:16,881 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 09:06:16,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 09:06:17,124 [I] DB_PATCHES have already been applied
I20260928 09:06:17.130663 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAWHStC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 09:06:17.130735 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 09:06:17.133944 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39331
Sep 28, 2026 9:06:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:06:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:06:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:06:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:06:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:06:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:06:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 09:06:18.251776 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.860s user 0.002s sys 0.006s
Sep 28, 2026 9:06:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:06:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:06:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:06:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:06:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 09:06:40.628021 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 09:06:40 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586400, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 09:06:40.718820 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 09:06:40.728864 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 09:06:41.652601 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 09:06:41.652678 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms
I20260928 09:06:41.652700 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50485
2026-09-28 09:06:41,815 [I] DB FLAVOR :POSTGRES
2026-09-28 09:06:41,815 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:06:41,815 [I] Checking connection
2026-09-28 09:06:41,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 09:06:42,062 [I] connection success
2026-09-28 09:06:42,062 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 09:06:42,062 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 09:06:42,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 09:06:42,327 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 09:06:42,327 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:06:42,328 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 09:06:42,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50485/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 09:06:42,579 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 09:06:43.071416 392 mini_ranger_kms.cc:310] Created kms service
I20260928 09:06:43.257437 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 09:06:43.311609 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 09:06:43 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586400, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:43.743065 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 09:06:43.743197 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAWHStC/build/release/bin/postgresql.jar:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAWHStC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 09:06:43.743219 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 09:06:43.746471 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12540
I20260928 09:06:43.746562 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39861
Sep 28, 2026 9:06:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:06:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:06:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:06:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:06:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:06:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 09:06:46.035616 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.317s user 0.009s sys 0.136s
I20260928 09:06:46.035768 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 09:06:46.035825 392 mini_ranger_kms.cc:401] 127.0.98.20:39861/kms/v1/keys
Sep 28 09:06:46 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586406, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 09:06:46 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586406, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:06:46 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586406, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:06:49 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:06:49.341837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38003
--ranger_config_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:49.425256 12591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:49.425524 12591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:49.425596 12591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:49.426949 12591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:49.427033 12591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:49.427060 12591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:49.427114 12591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:49.427145 12591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:49.428834 12591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38003
--ranger_config_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:49.429119 12591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:49.429353 12591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:49.435834 12597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:49.435930 12596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:49.436035 12591 server_base.cc:1034] running on GCE node
W20260928 09:06:49.435845 12599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:49.436416 12591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:49.436775 12591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:49.437942 12591 hybrid_clock.cc:648] HybridClock initialized: now 1790586409437928 us; error 28 us; skew 500 ppm
Sep 28 09:06:49 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:49.448024 12591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:06:49.448614 12591 webserver.cc:460] Webserver started at http://127.0.98.62:33019/ using document root <none> and password file <none>
I20260928 09:06:49.448882 12591 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:49.448926 12591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:49.449075 12591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:49 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:49.511785 12591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "da2156d0786c4502925725d1b0f9f992"
format_stamp: "Formatted at 2026-09-28 09:06:49 on dist-test-slave-smns"
server_key: "033abfe0d8d1c88983c2d595c4b8ff5c"
server_key_iv: "0d44bcf35ec24893b3748ff9cd0acee5"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:49.512256 12591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "da2156d0786c4502925725d1b0f9f992"
format_stamp: "Formatted at 2026-09-28 09:06:49 on dist-test-slave-smns"
server_key: "033abfe0d8d1c88983c2d595c4b8ff5c"
server_key_iv: "0d44bcf35ec24893b3748ff9cd0acee5"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:49.514146 12591 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 09:06:49.528276 12606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:49.528455 12591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:06:49.528540 12591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "da2156d0786c4502925725d1b0f9f992"
format_stamp: "Formatted at 2026-09-28 09:06:49 on dist-test-slave-smns"
server_key: "033abfe0d8d1c88983c2d595c4b8ff5c"
server_key_iv: "0d44bcf35ec24893b3748ff9cd0acee5"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:49.528637 12591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:49.550545 12591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:49.552708 12591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:49.552866 12591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:49.557917 12591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38003
I20260928 09:06:49.557991 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38003 every 8 connection(s)
I20260928 09:06:49.558367 12591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:06:49.558651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 12591
I20260928 09:06:49.558758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:06:49.558936 392 external_mini_cluster.cc:1428] Setting key 291095caf2fbe2a3a9e8ffbfee92d576
I20260928 09:06:49.559227 12659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:49.562511 12659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Bootstrap starting.
I20260928 09:06:49.563227 12659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:49.563530 12659 log.cc:824] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:49.564313 12659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: No bootstrap required, opened a new log
I20260928 09:06:49.567032 12659 raft_consensus.cc:348] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:49.567214 12659 raft_consensus.cc:374] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:49.567278 12659 raft_consensus.cc:729] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2156d0786c4502925725d1b0f9f992, State: Initialized, Role: FOLLOWER
Sep 28 09:06:49 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:49.567456 12659 consensus_queue.cc:260] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:49.567546 12659 raft_consensus.cc:388] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:06:49.567591 12659 raft_consensus.cc:482] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:06:49.567648 12659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:49.568449 12659 raft_consensus.cc:504] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:49.568564 12659 leader_election.cc:302] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da2156d0786c4502925725d1b0f9f992; no voters:
I20260928 09:06:49.568795 12659 leader_election.cc:288] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:06:49.569052 12659 sys_catalog.cc:564] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:49.570086 12664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:49.570297 12664 raft_consensus.cc:686] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 LEADER]: Becoming Leader. State: Replica: da2156d0786c4502925725d1b0f9f992, State: Running, Role: LEADER
I20260928 09:06:49.570438 12664 consensus_queue.cc:237] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:49.571344 12659 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 09:06:49.571940 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:49.559452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.0.98.62:38003)
0928 09:06:49.559682 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:49.559692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:49.559725 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:06:49.560400 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:49.560410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:49.560477 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:49.560628 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:49.561434 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:49.562895 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:49.564005 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:49.564615 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:49.564703 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:49.567889 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:49.567894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:49.567894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:49.567906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:49.569950 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:49.570422 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:49.570427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:49.570428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:49.570551 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:49.570810 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:49.570811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:49.570811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:49.571156 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:49.571288 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:49.571558 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 09:06:49.571701 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260928 09:06:49.571975 12664 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da2156d0786c4502925725d1b0f9f992. Latest consensus state: current_term: 1 leader_uuid: "da2156d0786c4502925725d1b0f9f992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } } }
I20260928 09:06:49.572038 12664 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:49.571954 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da2156d0786c4502925725d1b0f9f992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } } }
I20260928 09:06:49.573704 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:49.575212 12680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:06:49.576002 12680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:06:49.578755 12680 catalog_manager.cc:1347] Generated new cluster ID: cfe74750cb0f45a38fab452541236157
I20260928 09:06:49.578802 12680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:06:49.591220 12680 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:06:49.592499 12680 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:06:49.605247 12680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Generated new TSK 0
I20260928 09:06:49.605621 12680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:06:50.393502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38003
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:50.508045 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:50.508373 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:50.508423 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:50.510313 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:50.510385 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:50.510425 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:06:50.512691 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:50.512962 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:50.513279 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:50.519722 12710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:50.519804 12711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:50.519860 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:50.520104 12705 server_base.cc:1034] running on GCE node
I20260928 09:06:50.520324 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:50.520617 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:50.521792 12705 hybrid_clock.cc:648] HybridClock initialized: now 1790586410521761 us; error 47 us; skew 500 ppm
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:50.525063 12705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:06:50.525638 12705 webserver.cc:460] Webserver started at http://127.0.98.1:35167/ using document root <none> and password file <none>
I20260928 09:06:50.525949 12705 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:50.526021 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:50.526221 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:50.541606 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "72eaf9ced1b0436db9ebc2abc79a3767"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "c63e78ad4b98c26fadf6fe20643d296a"
server_key_iv: "09e4a7237404888c0cd03b237c753c9f"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:50.542235 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72eaf9ced1b0436db9ebc2abc79a3767"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "c63e78ad4b98c26fadf6fe20643d296a"
server_key_iv: "09e4a7237404888c0cd03b237c753c9f"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:50.548260 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260928 09:06:50.567363 12722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:50.567649 12705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 09:06:50.567762 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "72eaf9ced1b0436db9ebc2abc79a3767"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "c63e78ad4b98c26fadf6fe20643d296a"
server_key_iv: "09e4a7237404888c0cd03b237c753c9f"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:50.567862 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:50.590830 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:50.593456 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:50.593648 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:50.593921 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:50.594357 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:50.594455 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:50.594556 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:50.594640 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:50.601583 12705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44411
I20260928 09:06:50.602114 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:06:50.603041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 12705
I20260928 09:06:50.603150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:06:50.603361 392 external_mini_cluster.cc:1428] Setting key ec14528761b2e84587dcd40a4e170340
I20260928 09:06:50.614089 12835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44411 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:50.628890 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:50.614832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38003 (local address 127.0.98.1:32871)
0928 09:06:50.617616 (+ 2784us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:50.617673 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:50.618380 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:50.618873 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:50.618878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:50.619142 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:50.619445 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:50.619452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.620516 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.620517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:50.621862 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:50.621865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.622448 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.622449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:50.622594 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:50.623238 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:50.623250 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:50.624849 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:50.627038 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:50.627047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:50.627050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:50.627383 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:50.627633 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:50.627634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:50.627635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:50.627789 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:50.628108 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:50.628116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:50.628333 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 09:06:50.628449 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 09:06:50.628625 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2726,"thread_start_us":100,"threads_started":1}
I20260928 09:06:50.629312 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:50.615151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32871 (local address 127.0.98.62:38003)
0928 09:06:50.618603 (+ 3452us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:50.618606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:50.618615 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:06:50.618642 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:50.618643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:50.618657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:50.618721 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:50.619569 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.620397 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.621965 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.622301 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.622361 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:50.624965 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:50.624966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:50.624967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:50.624975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:50.626896 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:50.627478 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:50.627480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:50.627480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:50.627551 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:50.627864 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:50.627864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:50.627865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:50.628008 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:50.628079 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:50.629170 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0928 09:06:50.629245 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3428,"thread_start_us":68,"threads_started":1}
I20260928 09:06:50.629812 12836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38003
I20260928 09:06:50.629961 12836 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:50.630185 12836 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:50.630739 12623 ts_manager.cc:194] Registered new tserver with Master: 72eaf9ced1b0436db9ebc2abc79a3767 (127.0.98.1:44411)
I20260928 09:06:50.631553 12623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:06:50.642303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38003
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:50.746312 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:50.746547 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:50.746591 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:50.748003 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:50.748065 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:50.748111 12842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:06:50.750008 12842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:50.750311 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:50.750586 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:50.755899 12849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:50.755919 12852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:50.755988 12850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:50.756309 12842 server_base.cc:1034] running on GCE node
I20260928 09:06:50.756587 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:50.756887 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:50.758044 12842 hybrid_clock.cc:648] HybridClock initialized: now 1790586410758022 us; error 35 us; skew 500 ppm
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:50.760877 12842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:06:50.761389 12842 webserver.cc:460] Webserver started at http://127.0.98.2:41761/ using document root <none> and password file <none>
I20260928 09:06:50.761699 12842 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:50.761766 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:50.761905 12842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:50.778883 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3ab36def5724437800b04c3193943d0"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "753089aa063275839619fc64f13641fb"
server_key_iv: "324f1a2fc9eec559faf5ca06d552fd71"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:50.779441 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3ab36def5724437800b04c3193943d0"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "753089aa063275839619fc64f13641fb"
server_key_iv: "324f1a2fc9eec559faf5ca06d552fd71"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:50.784541 12842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260928 09:06:50.811314 12859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:50.811578 12842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:06:50.811669 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "c3ab36def5724437800b04c3193943d0"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "753089aa063275839619fc64f13641fb"
server_key_iv: "324f1a2fc9eec559faf5ca06d552fd71"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:50.811761 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:50.838972 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:50.847066 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:50.847333 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:50.847673 12842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:50.848169 12842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:50.848269 12842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:50.848349 12842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:50.848412 12842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:50.855748 12842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33445
I20260928 09:06:50.856314 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:06:50.860785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 12842
I20260928 09:06:50.860903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:06:50.861114 392 external_mini_cluster.cc:1428] Setting key 5f1aa3802c185fa9bc33d64edb1c6bd1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:06:50.878149 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33445 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 28 09:06:50 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:50.897009 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:50.879195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49447 (local address 127.0.98.62:38003)
0928 09:06:50.879245 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:50.879247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:50.879544 (+ 297us) server_negotiation.cc:407] Connection header received
0928 09:06:50.882201 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:50.882203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:50.882216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:50.882247 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:50.883128 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.883891 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.885345 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.885705 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.885774 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:50.892761 (+ 6987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:50.892764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:50.892764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:50.892773 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:50.894729 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:50.895647 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:50.895648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:50.895649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:50.895711 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:50.896128 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:50.896129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:50.896129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:50.896282 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:50.896328 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:50.896810 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 09:06:50.896908 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:06:50.897971 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:50.878888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38003 (local address 127.0.98.2:49447)
0928 09:06:50.879415 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:50.879461 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:50.882049 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:50.882448 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:50.882452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:50.882702 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:50.882998 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:50.883005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.884019 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.884020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:50.885241 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:50.885244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:50.885855 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:50.885856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:50.886003 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:50.886668 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:50.886678 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:50.892602 (+ 5924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:50.895178 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:50.895190 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:50.895193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:50.895541 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:50.895863 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:50.895864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:50.895864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:50.896033 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:50.896377 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:50.896384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:50.896639 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 09:06:50.897493 (+ 854us) client_negotiation.cc:240] Negotiation successful
0928 09:06:50.897720 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":93,"threads_started":1}
I20260928 09:06:50.898715 12974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38003
I20260928 09:06:50.898810 12974 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:50.898996 12974 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:50.899406 12623 ts_manager.cc:194] Registered new tserver with Master: c3ab36def5724437800b04c3193943d0 (127.0.98.2:33445)
I20260928 09:06:50.899956 12623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:06:50.907691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38003
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:51.029727 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:51.029987 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:51.030028 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:51.031328 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:51.031401 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:51.031446 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:06:51.033349 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:51.033923 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:51.034209 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:51.040107 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.040645 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.040831 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:51.041740 12979 server_base.cc:1034] running on GCE node
I20260928 09:06:51.042040 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:51.042321 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:51.045606 12979 hybrid_clock.cc:648] HybridClock initialized: now 1790586411045564 us; error 47 us; skew 500 ppm
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:51.048907 12979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:06:51.049520 12979 webserver.cc:460] Webserver started at http://127.0.98.3:46681/ using document root <none> and password file <none>
I20260928 09:06:51.049830 12979 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:51.049901 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:51.050038 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:51.063874 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a14c617937f43e1a645c2cdc59e9fcb"
format_stamp: "Formatted at 2026-09-28 09:06:51 on dist-test-slave-smns"
server_key: "475c52786244f6ac3a33f41592acada3"
server_key_iv: "099a7c35e37e87f0e3ab04b4ce848234"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.064359 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a14c617937f43e1a645c2cdc59e9fcb"
format_stamp: "Formatted at 2026-09-28 09:06:51 on dist-test-slave-smns"
server_key: "475c52786244f6ac3a33f41592acada3"
server_key_iv: "099a7c35e37e87f0e3ab04b4ce848234"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.069100 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260928 09:06:51.082119 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.082358 12979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:06:51.082438 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "6a14c617937f43e1a645c2cdc59e9fcb"
format_stamp: "Formatted at 2026-09-28 09:06:51 on dist-test-slave-smns"
server_key: "475c52786244f6ac3a33f41592acada3"
server_key_iv: "099a7c35e37e87f0e3ab04b4ce848234"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.082527 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:51.108284 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:51.110823 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:51.110977 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:51.111218 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:51.111562 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:51.111613 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.111655 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:51.111678 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.119438 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44281
I20260928 09:06:51.119517 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44281 every 8 connection(s)
I20260928 09:06:51.120029 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:06:51.126591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 12979
I20260928 09:06:51.126713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:06:51.126951 392 external_mini_cluster.cc:1428] Setting key 6d767852486edc861019de3fb8868789
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:51.131948 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.121085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55295 (local address 127.0.98.62:38003)
0928 09:06:51.121132 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:51.121135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:51.121334 (+ 199us) server_negotiation.cc:407] Connection header received
0928 09:06:51.122060 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:51.122062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:51.122075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:51.122105 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:51.122942 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.123646 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.125245 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.125532 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.125591 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.128645 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:51.128647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:51.128647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:51.128654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:51.130390 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.130889 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.130890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.130890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.130949 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.131207 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.131208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.131208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.131338 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:51.131377 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:51.131761 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 09:06:51.131853 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:06:51.132762 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.120797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38003 (local address 127.0.98.3:55295)
0928 09:06:51.121204 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:51.121250 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:51.121927 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:51.122299 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:51.122304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:51.122541 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:51.122830 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.122837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.123766 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.123766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.125137 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.125140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.125704 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.125705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.125842 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.126462 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:51.126472 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:51.128524 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:51.130492 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.130499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.130501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.130801 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.131015 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.131016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.131016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.131137 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.131419 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:51.131425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:51.131597 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 09:06:51.132313 (+ 716us) client_negotiation.cc:240] Negotiation successful
0928 09:06:51.132484 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":89,"threads_started":1}
I20260928 09:06:51.133412 13108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38003
I20260928 09:06:51.133502 13108 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.133711 13108 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:51.134162 12623 ts_manager.cc:194] Registered new tserver with Master: 6a14c617937f43e1a645c2cdc59e9fcb (127.0.98.3:44281)
I20260928 09:06:51.134639 12623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55295
I20260928 09:06:51.138460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:06:51.139125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 12705
I20260928 09:06:51.147614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 12842
I20260928 09:06:51.153427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 12979
I20260928 09:06:51.159574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 12591
I20260928 09:06:51.164386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38003
--webserver_interface=127.0.98.62
--webserver_port=33019
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38003
--ranger_config_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:51.246507 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:51.246757 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:51.246794 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:51.247964 13111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:51.248036 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:51.248064 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:51.248081 13111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:51.248116 13111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:51.249933 13111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38003
--ranger_config_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:51.250211 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:51.250437 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:51.256172 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.256181 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.256258 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:51.256366 13111 server_base.cc:1034] running on GCE node
I20260928 09:06:51.256642 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:51.256960 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:51.258133 13111 hybrid_clock.cc:648] HybridClock initialized: now 1790586411258117 us; error 33 us; skew 500 ppm
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:51.261330 13111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:06:51.261927 13111 webserver.cc:460] Webserver started at http://127.0.98.62:33019/ using document root <none> and password file <none>
I20260928 09:06:51.262168 13111 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:51.262235 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:51.312814 13111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:06:51.313630 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.313813 13111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:06:51.313905 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "da2156d0786c4502925725d1b0f9f992"
format_stamp: "Formatted at 2026-09-28 09:06:49 on dist-test-slave-smns"
server_key: "033abfe0d8d1c88983c2d595c4b8ff5c"
server_key_iv: "0d44bcf35ec24893b3748ff9cd0acee5"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.314311 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:51.332293 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:51.334884 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:51.335139 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:51.340138 13111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38003
I20260928 09:06:51.340163 13179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38003 every 8 connection(s)
I20260928 09:06:51.340588 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:06:51.341971 13180 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:06:51.342646 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Bootstrap starting.
I20260928 09:06:51.350299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13111
I20260928 09:06:51.350625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44411
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38003
--webserver_port=35167
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:06:51.351667 13180 log.cc:824] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:51.354061 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:06:51.354415 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Bootstrap complete.
I20260928 09:06:51.359220 13180 raft_consensus.cc:348] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:51.359685 13180 raft_consensus.cc:729] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2156d0786c4502925725d1b0f9f992, State: Initialized, Role: FOLLOWER
I20260928 09:06:51.359938 13180 consensus_queue.cc:260] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:51.360055 13180 raft_consensus.cc:388] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:06:51.360110 13180 raft_consensus.cc:482] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:06:51.360177 13180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:06:51.361933 13180 raft_consensus.cc:504] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:51.362078 13180 leader_election.cc:302] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da2156d0786c4502925725d1b0f9f992; no voters:
I20260928 09:06:51.362269 13180 leader_election.cc:288] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:06:51.362344 13184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:06:51.362498 13184 raft_consensus.cc:686] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [term 2 LEADER]: Becoming Leader. State: Replica: da2156d0786c4502925725d1b0f9f992, State: Running, Role: LEADER
I20260928 09:06:51.362578 13180 sys_catalog.cc:564] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:51.362588 13184 consensus_queue.cc:237] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } }
I20260928 09:06:51.362882 13184 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da2156d0786c4502925725d1b0f9f992. Latest consensus state: current_term: 2 leader_uuid: "da2156d0786c4502925725d1b0f9f992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } } }
I20260928 09:06:51.362941 13184 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:51.363061 13185 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da2156d0786c4502925725d1b0f9f992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2156d0786c4502925725d1b0f9f992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38003 } } }
I20260928 09:06:51.363131 13188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:06:51.363196 13185 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:51.363670 13188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:06:51.363840 13188 catalog_manager.cc:1259] Loaded cluster ID: cfe74750cb0f45a38fab452541236157
I20260928 09:06:51.363884 13188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:06:51.364377 13180 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 09:06:51.365074 13188 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:06:51.365262 13188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Loaded TSK: 0
I20260928 09:06:51.365473 13188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:06:51.453936 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:51.454243 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:51.454293 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:51.456697 13182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:51.456780 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:51.456844 13182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:06:51.459446 13182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35167
--tserver_master_addrs=127.0.98.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:51.459725 13182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:51.460014 13182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:51.466603 13218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.467741 13217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.467938 13220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:51.468636 13182 server_base.cc:1034] running on GCE node
I20260928 09:06:51.468869 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:51.469163 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:51.470342 13182 hybrid_clock.cc:648] HybridClock initialized: now 1790586411470309 us; error 48 us; skew 500 ppm
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:51.473799 13182 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:06:51.474355 13182 webserver.cc:460] Webserver started at http://127.0.98.1:35167/ using document root <none> and password file <none>
I20260928 09:06:51.474617 13182 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:51.474701 13182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:51.498019 13182 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260928 09:06:51.499118 13228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.499284 13182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:06:51.499368 13182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "72eaf9ced1b0436db9ebc2abc79a3767"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "c63e78ad4b98c26fadf6fe20643d296a"
server_key_iv: "09e4a7237404888c0cd03b237c753c9f"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.499871 13182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:51.527364 13182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:51.529891 13182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:51.530046 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:51.530292 13182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:51.530637 13182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:51.530682 13182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.530712 13182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:51.530738 13182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.536916 13182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44411
I20260928 09:06:51.537361 13182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:06:51.541085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13182
I20260928 09:06:51.541424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33445
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38003
--webserver_port=41761
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:06:51.549998 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44411 every 8 connection(s)
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:51.566181 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.550860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38003 (local address 127.0.98.1:36197)
0928 09:06:51.552140 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:51.552202 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:51.553012 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:51.553920 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:51.553924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:51.554181 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:51.554448 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.554454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.556539 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.556540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.557879 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.557882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.558818 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.558819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.558954 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.559622 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:51.559634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:51.561350 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:51.564201 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.564208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.564210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.564492 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.564793 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.564794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.564794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.564939 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.565427 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:51.565433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:51.565624 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 09:06:51.565748 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 09:06:51.565897 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":98,"threads_started":1}
I20260928 09:06:51.566205 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.551309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36197 (local address 127.0.98.62:38003)
0928 09:06:51.553259 (+ 1950us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:51.553270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:51.553314 (+ 44us) server_negotiation.cc:407] Connection header received
0928 09:06:51.553484 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:51.553492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:51.553591 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:51.553750 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:51.554594 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.556407 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.557982 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.558702 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.558833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.561475 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:51.561480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:51.561480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:51.561494 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:51.564081 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.564582 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.564587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.564587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.564711 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.565017 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.565018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.565019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.565344 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:51.565443 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:51.565835 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 09:06:51.565969 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":93,"threads_started":1}
I20260928 09:06:51.567816 13342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38003
I20260928 09:06:51.567925 13342 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.568100 13342 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
W20260928 09:06:51.569003 13342 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38003 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 09:06:51.569070 13342 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.569125 13342 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:51.569876 13342 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.569958 13342 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
W20260928 09:06:51.570320 13342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:06:51.643337 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:51.643623 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:51.643662 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:51.645403 13344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:51.645447 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:51.645486 13344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:06:51.647648 13344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41761
--tserver_master_addrs=127.0.98.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:51.647895 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:51.648142 13344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:51.654196 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.654188 13354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.654186 13352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:51.654747 13344 server_base.cc:1034] running on GCE node
I20260928 09:06:51.654948 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:51.655217 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:51.656388 13344 hybrid_clock.cc:648] HybridClock initialized: now 1790586411656372 us; error 34 us; skew 500 ppm
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:51.659590 13344 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:06:51.660136 13344 webserver.cc:460] Webserver started at http://127.0.98.2:41761/ using document root <none> and password file <none>
I20260928 09:06:51.660404 13344 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:51.660477 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:51.687616 13344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 09:06:51.688498 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.688694 13344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:06:51.688774 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "c3ab36def5724437800b04c3193943d0"
format_stamp: "Formatted at 2026-09-28 09:06:50 on dist-test-slave-smns"
server_key: "753089aa063275839619fc64f13641fb"
server_key_iv: "324f1a2fc9eec559faf5ca06d552fd71"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.689225 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:51.701280 13344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:51.703886 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:51.704051 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:51.704273 13344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:51.704667 13344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:51.704705 13344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.704747 13344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:51.704774 13344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.711894 13344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33445
I20260928 09:06:51.712369 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:06:51.714062 13475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33445 every 8 connection(s)
I20260928 09:06:51.719952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13344
I20260928 09:06:51.720304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44281
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38003
--webserver_port=46681
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:51.726797 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.716462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38091 (local address 127.0.98.62:38003)
0928 09:06:51.716515 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:51.716518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:51.716532 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:06:51.717153 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:51.717155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:51.717168 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:51.717249 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:51.718014 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.718886 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.720456 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.720767 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.720810 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.723625 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:51.723627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:51.723628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:51.723637 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:51.725246 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.725784 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.725785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.725786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.725846 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.726112 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.726113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.726113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.726240 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:51.726310 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:51.726618 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 09:06:51.726703 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:06:51.727013 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.716128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38003 (local address 127.0.98.2:38091)
0928 09:06:51.716342 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:51.716398 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:51.717014 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:51.717341 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:51.717346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:51.717614 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:51.717893 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.717899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.719008 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.719009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.720334 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.720336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.720916 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.720917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.721058 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.721761 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:51.721772 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:51.723483 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:51.725368 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.725375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.725378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.725692 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.725913 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.725914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.725914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.726038 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.726313 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:51.726321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:51.726481 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 09:06:51.726585 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 09:06:51.726750 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260928 09:06:51.727569 13476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38003
I20260928 09:06:51.727691 13476 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.727903 13476 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
W20260928 09:06:51.733476 13476 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38003 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 09:06:51.733669 13476 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.733793 13476 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:51.734180 13476 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.734305 13476 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
W20260928 09:06:51.734642 13476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:06:51.841645 13479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:51.841938 13479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:51.841984 13479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:51.843863 13479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:51.843935 13479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:51.843981 13479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:06:51.846390 13479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46681
--tserver_master_addrs=127.0.98.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:51.846668 13479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:51.846985 13479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:51.855672 13487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.856001 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:51.858438 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:51.861789 13479 server_base.cc:1034] running on GCE node
I20260928 09:06:51.862114 13479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:51.862572 13479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:51.863788 13479 hybrid_clock.cc:648] HybridClock initialized: now 1790586411863774 us; error 52 us; skew 500 ppm
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:51.867513 13479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:06:51.868139 13479 webserver.cc:460] Webserver started at http://127.0.98.3:46681/ using document root <none> and password file <none>
I20260928 09:06:51.868475 13479 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:51.868595 13479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:51.890974 13479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260928 09:06:51.891871 13495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.892069 13479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:06:51.892164 13479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "6a14c617937f43e1a645c2cdc59e9fcb"
format_stamp: "Formatted at 2026-09-28 09:06:51 on dist-test-slave-smns"
server_key: "475c52786244f6ac3a33f41592acada3"
server_key_iv: "099a7c35e37e87f0e3ab04b4ce848234"
server_key_version: "kuduclusterkey@0"
I20260928 09:06:51.892554 13479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:51.908970 13479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:51.911603 13479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:51.911783 13479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:51.912025 13479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:51.912441 13479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:51.912495 13479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.912557 13479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:51.912581 13479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:51.919050 13479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44281
I20260928 09:06:51.919484 13479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:06:51.920966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13479
I20260928 09:06:51.926586 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44281 every 8 connection(s)
I20260928 09:06:51.929162 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.921477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.0.98.62:38003)
0928 09:06:51.921524 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:51.921526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:51.921705 (+ 179us) server_negotiation.cc:407] Connection header received
0928 09:06:51.921797 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:51.921798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:51.921809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:51.921883 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:51.922496 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.922991 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.923760 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.923984 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.924073 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.925640 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:51.925643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:51.925643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:51.925652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:51.927385 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.928272 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.928274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.928274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.928338 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.928554 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.928555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.928556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.928695 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:51.928875 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:51.928950 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 09:06:51.929053 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 09:06:51 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:51.938751 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.927824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38003 (local address 127.0.98.3:59327)
0928 09:06:51.928037 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:51.928082 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:51.928998 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:51.929391 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:51.929397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:51.929747 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:51.930066 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.930073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.930876 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.930877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.931945 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:51.931947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.932417 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.932418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:51.932565 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.933128 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:51.933137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:51.934694 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:51.936896 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.936904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.936906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.937228 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.937460 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:51.937460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:51.937461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:51.937663 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:51.937992 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:51.937998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:51.938200 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 09:06:51.938308 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 09:06:51.938476 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":131,"threads_started":1}
I20260928 09:06:51.939914 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:51.928176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59327 (local address 127.0.98.62:38003)
0928 09:06:51.928455 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:51.928457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:51.928464 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:06:51.929167 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:51.929168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:51.929183 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:51.929250 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:51.930161 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.930782 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.932067 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:51.932297 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:51.932397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:51.934817 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:51.934818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:51.934819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:51.934826 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:51.936789 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.937321 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.937322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.937322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.937388 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:51.937748 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:51.937750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:51.937750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:51.937898 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:51.937965 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:51.939705 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0928 09:06:51.939802 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":60,"threads_started":1}
I20260928 09:06:51.940289 13610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38003
I20260928 09:06:51.940407 13610 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.940627 13610 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
W20260928 09:06:51.941444 13610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38003 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 09:06:51.941538 13610 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.941640 13610 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:51.941987 13610 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:51.942083 13610 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
W20260928 09:06:51.942449 13610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 09:06:52 dist-test-slave-smns krb5kdc[12147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:52.570515 13342 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:52.570679 13342 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:52.571137 13144 ts_manager.cc:194] Registered new tserver with Master: 72eaf9ced1b0436db9ebc2abc79a3767 (127.0.98.1:44411)
I20260928 09:06:52.572216 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36197
I20260928 09:06:52.734848 13476 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:52.735015 13476 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:52.735360 13143 ts_manager.cc:194] Registered new tserver with Master: c3ab36def5724437800b04c3193943d0 (127.0.98.2:33445)
I20260928 09:06:52.735908 13143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38091
Sep 28 09:06:52 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:52.942642 13610 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:52.942826 13610 heartbeater.cc:511] Master 127.0.98.62:38003 requested a full tablet report, sending...
I20260928 09:06:52.943148 13143 ts_manager.cc:194] Registered new tserver with Master: 6a14c617937f43e1a645c2cdc59e9fcb (127.0.98.3:44281)
I20260928 09:06:52.943696 13143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59327
I20260928 09:06:52.950487 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:06:52.959519 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.951480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.98.62:38003)
0928 09:06:52.951594 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:52.951596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:52.951607 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:06:52.951663 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:52.951664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:52.951677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:52.951760 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:52.952425 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.953194 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.954275 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.954588 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.954679 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.956147 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:52.956148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:52.956149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:52.956158 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:52.958023 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:52.958413 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:52.958414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:52.958414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:52.958482 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:52.958700 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:52.958700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:52.958701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:52.958840 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:52.958944 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:52.959329 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 09:06:52.959413 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 09:06:52.961956 13143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:06:52.964447 13143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:06:52.973192 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.968023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33445 (local address 127.0.0.1:43088)
0928 09:06:52.968929 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:52.968937 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:52.969015 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:52.969253 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:52.969254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:52.969256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:52.969298 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:52.969299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.970592 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.970593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:52.971890 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:52.971893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.972907 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.972907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:52.972987 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.972999 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 09:06:52.973060 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:06:52.973108 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260928 09:06:52.973503 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.968117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.0.98.2:33445)
0928 09:06:52.968237 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:52.968242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:52.968988 (+ 746us) server_negotiation.cc:407] Connection header received
0928 09:06:52.969090 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:52.969093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:52.969107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:52.969155 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:52.969367 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.970490 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.971994 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.972728 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.972775 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.972846 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:52.973381 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 09:06:52.973427 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260928 09:06:52.973943 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.967800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44411 (local address 127.0.0.1:38260)
0928 09:06:52.968617 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:52.968625 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:52.968662 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:52.968857 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:52.968860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:52.968863 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:52.968939 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:52.968941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.970224 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.970225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:52.971887 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:52.971890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.973203 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.973203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:52.973517 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.973522 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:06:52.973825 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 09:06:52.973868 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":37,"thread_start_us":93,"threads_started":1}
I20260928 09:06:52.974267 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.968215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.0.98.1:44411)
0928 09:06:52.968330 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:52.968334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:52.968714 (+ 380us) server_negotiation.cc:407] Connection header received
0928 09:06:52.968751 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:52.968754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:52.968769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:52.968837 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:52.969023 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.970127 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.971991 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.973030 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.973070 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.973144 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:52.974146 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0928 09:06:52.974193 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
I20260928 09:06:52.974267 13410 tablet_service.cc:1467] Processing CreateTablet for tablet 668c8776806049d1bee574f9cae7e0df (DEFAULT_TABLE table=test-table [id=b32fedf1644a4426920bc8c5f1216162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:06:52.974658 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668c8776806049d1bee574f9cae7e0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:52.975122 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 668c8776806049d1bee574f9cae7e0df (DEFAULT_TABLE table=test-table [id=b32fedf1644a4426920bc8c5f1216162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:06:52.975431 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668c8776806049d1bee574f9cae7e0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:52.981514 13643 tablet_bootstrap.cc:492] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Bootstrap starting.
I20260928 09:06:52.981604 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.967799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44281 (local address 127.0.0.1:60578)
0928 09:06:52.968387 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:52.968427 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:52.968512 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:52.977859 (+ 9347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:52.977860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:52.977864 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:52.977911 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:52.977912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.979269 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.979270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:52.980386 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:52.980389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.981317 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.981318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:52.981383 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.981387 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:06:52.981449 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:06:52.981489 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":74,"threads_started":1}
I20260928 09:06:52.982405 13643 tablet_bootstrap.cc:654] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:52.982601 13644 tablet_bootstrap.cc:492] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: Bootstrap starting.
I20260928 09:06:52.983206 13644 tablet_bootstrap.cc:654] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:52.983508 13644 log.cc:824] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:52.984058 13643 log.cc:824] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:52.984616 13644 tablet_bootstrap.cc:492] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: No bootstrap required, opened a new log
I20260928 09:06:52.984686 13644 ts_tablet_manager.cc:1397] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:06:52.987290 13644 raft_consensus.cc:348] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:52.987468 13644 raft_consensus.cc:374] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:52.987498 13644 raft_consensus.cc:729] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72eaf9ced1b0436db9ebc2abc79a3767, State: Initialized, Role: FOLLOWER
I20260928 09:06:52.987977 13643 tablet_bootstrap.cc:492] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: No bootstrap required, opened a new log
I20260928 09:06:52.988051 13643 ts_tablet_manager.cc:1397] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 09:06:52.989674 13644 consensus_queue.cc:260] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:52.989809 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:52.968047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.98.3:44281)
0928 09:06:52.977677 (+ 9630us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:52.977683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:52.977695 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:06:52.977730 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:52.977732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:52.977748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:52.977975 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:52.978037 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.979164 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.980476 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:52.981204 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:52.981292 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:52.981361 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:52.989649 (+ 8288us) server_negotiation.cc:299] Negotiation successful
0928 09:06:52.989718 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9589,"thread_start_us":73,"threads_started":1}
I20260928 09:06:52.990237 13644 ts_tablet_manager.cc:1428] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260928 09:06:52.990423 13342 heartbeater.cc:503] Master 127.0.98.62:38003 was elected leader, sending a full tablet report...
I20260928 09:06:52.990500 13541 tablet_service.cc:1467] Processing CreateTablet for tablet 668c8776806049d1bee574f9cae7e0df (DEFAULT_TABLE table=test-table [id=b32fedf1644a4426920bc8c5f1216162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:06:52.990535 13643 raft_consensus.cc:348] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:52.990710 13643 raft_consensus.cc:374] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:52.990756 13643 raft_consensus.cc:729] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ab36def5724437800b04c3193943d0, State: Initialized, Role: FOLLOWER
I20260928 09:06:52.990864 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668c8776806049d1bee574f9cae7e0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:52.990866 13643 consensus_queue.cc:260] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:52.991101 13643 ts_tablet_manager.cc:1428] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 09:06:52.991209 13476 heartbeater.cc:503] Master 127.0.98.62:38003 was elected leader, sending a full tablet report...
I20260928 09:06:52.995771 13649 tablet_bootstrap.cc:492] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: Bootstrap starting.
I20260928 09:06:52.996456 13649 tablet_bootstrap.cc:654] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:52.996783 13649 log.cc:824] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:52.998240 13649 tablet_bootstrap.cc:492] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: No bootstrap required, opened a new log
I20260928 09:06:52.998323 13649 ts_tablet_manager.cc:1397] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 09:06:53.000937 13649 raft_consensus.cc:348] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:53.001122 13649 raft_consensus.cc:374] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:53.001186 13649 raft_consensus.cc:729] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a14c617937f43e1a645c2cdc59e9fcb, State: Initialized, Role: FOLLOWER
I20260928 09:06:53.001463 13649 consensus_queue.cc:260] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:53.001893 13610 heartbeater.cc:503] Master 127.0.98.62:38003 was elected leader, sending a full tablet report...
I20260928 09:06:53.001875 13649 ts_tablet_manager.cc:1428] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260928 09:06:53.051551 13343 tablet.cc:2369] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 09:06:53 dist-test-slave-smns krb5kdc[12147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:06:53.099910 13651 raft_consensus.cc:482] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:06:53.100090 13651 raft_consensus.cc:504] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:53.100555 13651 leader_election.cc:288] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72eaf9ced1b0436db9ebc2abc79a3767 (127.0.98.1:44411), c3ab36def5724437800b04c3193943d0 (127.0.98.2:33445)
I20260928 09:06:53.105315 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.100754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59071 (local address 127.0.98.1:44411)
0928 09:06:53.101661 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:53.101663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:53.101674 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:06:53.101705 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:53.101706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:53.101718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:53.101779 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:53.102023 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.102779 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.104302 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.104898 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.104951 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:53.104963 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:53.105195 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 09:06:53.105241 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876}
I20260928 09:06:53.105786 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.100698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44411 (local address 127.0.98.3:59071)
0928 09:06:53.100942 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:53.100954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:53.101020 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:53.101809 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:53.101810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:53.101813 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:53.101912 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:53.101915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.102888 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.102889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:53.104190 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:53.104193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.105027 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.105027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:53.105096 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:53.105101 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:06:53.105632 (+ 531us) client_negotiation.cc:240] Negotiation successful
0928 09:06:53.105687 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20260928 09:06:53.107299 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c8776806049d1bee574f9cae7e0df" candidate_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" is_pre_election: true
I20260928 09:06:53.107497 13296 raft_consensus.cc:2454] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a14c617937f43e1a645c2cdc59e9fcb in term 0.
I20260928 09:06:53.107746 13499 leader_election.cc:302] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a14c617937f43e1a645c2cdc59e9fcb, 72eaf9ced1b0436db9ebc2abc79a3767; no voters:
I20260928 09:06:53.107889 13651 raft_consensus.cc:2781] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:06:53.107928 13651 raft_consensus.cc:482] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:06:53.107949 13651 raft_consensus.cc:3037] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:53.108822 13651 raft_consensus.cc:504] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:53.108963 13651 leader_election.cc:288] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [CANDIDATE]: Term 1 election: Requested vote from peers 72eaf9ced1b0436db9ebc2abc79a3767 (127.0.98.1:44411), c3ab36def5724437800b04c3193943d0 (127.0.98.2:33445)
I20260928 09:06:53.109126 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c8776806049d1bee574f9cae7e0df" candidate_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72eaf9ced1b0436db9ebc2abc79a3767"
I20260928 09:06:53.109221 13296 raft_consensus.cc:3037] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:53.109593 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.104660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51745 (local address 127.0.98.2:33445)
0928 09:06:53.104707 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:53.104709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:53.106145 (+ 1436us) server_negotiation.cc:407] Connection header received
0928 09:06:53.106263 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:53.106264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:53.106276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:53.106305 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:06:53.106469 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.107176 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.108428 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.108856 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.108892 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:53.108901 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:53.109440 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 09:06:53.109506 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:06:53.109779 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.101925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33445 (local address 127.0.98.3:51745)
0928 09:06:53.106096 (+ 4171us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:53.106106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:53.106187 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:53.106366 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:53.106366 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:53.106368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:06:53.106401 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:53.106402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.107283 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.107283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:53.108325 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:53.108329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.109271 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.109272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:53.109337 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:53.109340 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:06:53.109675 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 09:06:53.109714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4158,"thread_start_us":77,"threads_started":1}
I20260928 09:06:53.110181 13296 raft_consensus.cc:2454] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a14c617937f43e1a645c2cdc59e9fcb in term 1.
I20260928 09:06:53.110522 13430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c8776806049d1bee574f9cae7e0df" candidate_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ab36def5724437800b04c3193943d0" is_pre_election: true
I20260928 09:06:53.110495 13429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c8776806049d1bee574f9cae7e0df" candidate_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ab36def5724437800b04c3193943d0"
I20260928 09:06:53.110656 13430 raft_consensus.cc:2454] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a14c617937f43e1a645c2cdc59e9fcb in term 0.
I20260928 09:06:53.110821 13499 leader_election.cc:302] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a14c617937f43e1a645c2cdc59e9fcb, 72eaf9ced1b0436db9ebc2abc79a3767; no voters:
I20260928 09:06:53.110929 13651 raft_consensus.cc:2781] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:53.111106 13651 raft_consensus.cc:686] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [term 1 LEADER]: Becoming Leader. State: Replica: 6a14c617937f43e1a645c2cdc59e9fcb, State: Running, Role: LEADER
I20260928 09:06:53.111209 13651 consensus_queue.cc:237] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } }
I20260928 09:06:53.112052 13143 catalog_manager.cc:5495] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a14c617937f43e1a645c2cdc59e9fcb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a14c617937f43e1a645c2cdc59e9fcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 } health_report { overall_health: UNKNOWN } } }
I20260928 09:06:53.132893 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.130092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.3:44281)
0928 09:06:53.130161 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:53.130162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:53.130171 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:06:53.130219 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:53.130220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:53.130246 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:53.130321 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:06:53.130548 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.131089 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.131940 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:53.132216 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:53.132295 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:53.132463 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:06:53.132611 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:06:53.132731 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:53.132750 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 09:06:53.132771 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:06:53.136401 13429 raft_consensus.cc:3037] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:53.137746 13429 raft_consensus.cc:1261] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 1 FOLLOWER]: Refusing update from remote peer 6a14c617937f43e1a645c2cdc59e9fcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:06:53.138095 13296 raft_consensus.cc:1261] T 668c8776806049d1bee574f9cae7e0df P 72eaf9ced1b0436db9ebc2abc79a3767 [term 1 FOLLOWER]: Refusing update from remote peer 6a14c617937f43e1a645c2cdc59e9fcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:06:53.138319 13655 consensus_queue.cc:1035] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3ab36def5724437800b04c3193943d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:06:53.138674 13655 consensus_queue.cc:1035] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72eaf9ced1b0436db9ebc2abc79a3767" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:06:53.147588 13658 mvcc.cc:205] Tried to move back new op lower bound from 7334241948199223296 to 7334241948104249344. Current Snapshot: MvccSnapshot[applied={T|T < 7334241948199223296}]
I20260928 09:06:53.150238 13660 mvcc.cc:205] Tried to move back new op lower bound from 7334241948199223296 to 7334241948104249344. Current Snapshot: MvccSnapshot[applied={T|T < 7334241948199223296}]
I20260928 09:06:53.150381 13659 mvcc.cc:205] Tried to move back new op lower bound from 7334241948199223296 to 7334241948104249344. Current Snapshot: MvccSnapshot[applied={T|T < 7334241948199223296}]
I20260928 09:06:53.153014 13144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:06:53.153131 13144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:06:53.153916 13144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da2156d0786c4502925725d1b0f9f992: Sending DeleteTablet for 3 replicas of tablet 668c8776806049d1bee574f9cae7e0df
I20260928 09:06:53.154428 13541 tablet_service.cc:1514] Processing DeleteTablet for tablet 668c8776806049d1bee574f9cae7e0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:06:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60578
I20260928 09:06:53.157862 13410 tablet_service.cc:1514] Processing DeleteTablet for tablet 668c8776806049d1bee574f9cae7e0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:06:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43088
I20260928 09:06:53.157862 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 668c8776806049d1bee574f9cae7e0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:06:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38260
I20260928 09:06:53.158296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13182
I20260928 09:06:53.160100 13671 tablet_replica.cc:330] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: stopping tablet replica
I20260928 09:06:53.160220 13671 raft_consensus.cc:2229] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:06:53.160331 13671 raft_consensus.cc:2258] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:06:53.161048 13671 ts_tablet_manager.cc:1905] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 09:06:53.165874 13130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44411 (error 108)
I20260928 09:06:53.165998 13671 ts_tablet_manager.cc:1918] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:06:53.166069 13671 log.cc:1196] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/668c8776806049d1bee574f9cae7e0df
I20260928 09:06:53.166324 13671 ts_tablet_manager.cc:1939] T 668c8776806049d1bee574f9cae7e0df P c3ab36def5724437800b04c3193943d0: Deleting consensus metadata
I20260928 09:06:53.166479 13669 tablet_replica.cc:330] T 668c8776806049d1bee574f9cae7e0df P 6a14c617937f43e1a645c2cdc59e9fcb: stopping tablet replica
I20260928 09:06:53.166733 13129 catalog_manager.cc:4841] TS c3ab36def5724437800b04c3193943d0 (127.0.98.2:33445): tablet 668c8776806049d1bee574f9cae7e0df (table test-table [id=b32fedf1644a4426920bc8c5f1216162]) successfully deleted
I20260928 09:06:53.166819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13344
I20260928 09:06:53.173525 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.173352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44411 (local address 127.0.0.1:38268)
0928 09:06:53.173433 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:53.173486 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 09:06:53.173985 13130 catalog_manager.cc:4584] TS 72eaf9ced1b0436db9ebc2abc79a3767 (127.0.98.1:44411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 668c8776806049d1bee574f9cae7e0df: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44411: connect: Connection refused (error 111)
I20260928 09:06:53.176946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13479
I20260928 09:06:53.186378 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:53.186250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44281 (local address 127.0.0.1:60594)
0928 09:06:53.186306 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:53.186332 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:06:53.186474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13111
I20260928 09:06:53.194577 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 09:06:53.539141 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 09:06:53.539232 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 09:06:53.884007 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 09:06:53.884 UTC [12177] LOG: received smart shutdown request
2026-09-28 09:06:53.888 UTC [12177] LOG: background worker "logical replication launcher" (PID 12186) exited with exit code 1
2026-09-28 09:06:53.888 UTC [12181] LOG: shutting down
2026-09-28 09:06:54.088 UTC [12177] LOG: database system is shut down
2026-09-28T09:06:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49686 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:06:54 dist-test-slave-smns krb5kdc[13681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:06:54 dist-test-slave-smns krb5kdc[13681](info): set up 2 sockets
Sep 28 09:06:54 dist-test-slave-smns krb5kdc[13681](info): commencing operation
krb5kdc: starting...
W20260928 09:06:55.249516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:06:55.373718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38371,127.0.98.61:40565,127.0.98.60:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:55.456166 13694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:55.456369 13694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:55.456391 13694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:55.457602 13694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:55.457647 13694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:55.457662 13694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:55.457674 13694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:55.457687 13694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:55.459235 13694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38371,127.0.98.61:40565,127.0.98.60:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:55.459491 13694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:55.459731 13694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:55.464541 13702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:55.464536 13700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:55.464586 13699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:55.464854 13694 server_base.cc:1034] running on GCE node
I20260928 09:06:55.465093 13694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:55.465417 13694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:55.466596 13694 hybrid_clock.cc:648] HybridClock initialized: now 1790586415466573 us; error 44 us; skew 500 ppm
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:55.481173 13694 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:06:55.481758 13694 webserver.cc:460] Webserver started at http://127.0.98.62:44357/ using document root <none> and password file <none>
I20260928 09:06:55.482007 13694 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:55.482087 13694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:55.482239 13694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:06:55.483347 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "b8d5e302e5cc4102866483b8721d9082"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "37d519099431c8978bf2588e770ce615"
server_key_iv: "165d1d9f4513be245ebe07adcd71f658"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.483711 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8d5e302e5cc4102866483b8721d9082"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "37d519099431c8978bf2588e770ce615"
server_key_iv: "165d1d9f4513be245ebe07adcd71f658"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.485280 13694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:06:55.486052 13709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:55.486238 13694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:55.486341 13694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "b8d5e302e5cc4102866483b8721d9082"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "37d519099431c8978bf2588e770ce615"
server_key_iv: "165d1d9f4513be245ebe07adcd71f658"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.486433 13694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:55.511432 13694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:55.513355 13694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:55.513521 13694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:55.517639 13694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38371
I20260928 09:06:55.517676 13761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38371 every 8 connection(s)
I20260928 09:06:55.518050 13694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:06:55.518877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13694
I20260928 09:06:55.518926 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:55.518990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:06:55.519223 392 external_mini_cluster.cc:1428] Setting key 1dff3323be1be2bda1d872a45d26cc3f
I20260928 09:06:55.520614 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:55.534502 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.521441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38371 (local address 127.0.0.1:35030)
0928 09:06:55.521662 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.521700 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.522201 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.522519 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.522522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.522881 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.523039 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.523043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.524659 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.524660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.525348 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.525349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.526097 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.526098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.526158 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.526734 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.526742 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.530877 (+ 4135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.532794 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.532801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.532803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.533012 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.533269 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.533270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.533271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.533354 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.533806 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.533811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.533884 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.534084 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.534323 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":93,"thread_start_us":117,"threads_started":1}
I20260928 09:06:55.534502 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.521448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.0.98.62:38371)
0928 09:06:55.521665 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.521675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.521828 (+ 153us) server_negotiation.cc:407] Connection header received
0928 09:06:55.522373 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.522376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.522399 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.522506 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.523160 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.524544 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.525447 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.526005 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.526131 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.530979 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.530987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.530987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.530994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.532680 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.533109 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.533112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.533112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.533196 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.533421 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.533422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.533422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.533721 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.533825 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.534122 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.534245 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"spinlock_wait_cycles":8576,"thread_start_us":109,"threads_started":1}
I20260928 09:06:55.535300 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:06:55.535669 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.535550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33336)
0928 09:06:55.535597 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.535640 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:06:55.535827 13711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
I20260928 09:06:55.536290 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.536198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33342)
0928 09:06:55.536244 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.536268 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:06:55.536439 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40565: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 09:06:55.546579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40565
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38371,127.0.98.61:40565,127.0.98.60:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:06:55.585624 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } attempt: 1
I20260928 09:06:55.586200 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.586035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33352)
0928 09:06:55.586103 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.586148 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 09:06:55.586570 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.586482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33364)
0928 09:06:55.586526 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.586553 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:06:55.586692 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40565: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
W20260928 09:06:55.629046 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:55.629272 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:55.629297 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:55.630582 13769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:55.630650 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:55.630698 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:55.630730 13769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:55.630750 13769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:55.632318 13769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38371,127.0.98.61:40565,127.0.98.60:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:55.632588 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:55.632869 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:55.637761 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:55.637761 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:55.637938 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:55.638028 13769 server_base.cc:1034] running on GCE node
I20260928 09:06:55.638286 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:55.638578 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:55.639763 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790586415639757 us; error 52 us; skew 500 ppm
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:55.642382 13769 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 09:06:55.642856 13769 webserver.cc:460] Webserver started at http://127.0.98.61:33977/ using document root <none> and password file <none>
I20260928 09:06:55.643096 13769 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:55.643167 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:55.643319 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:06:55.644402 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data/instance:
uuid: "674345a3abc24d99be9bf1909d4d0978"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "a21d0c6b9d2d5f8816d2f50011071ba5"
server_key_iv: "92bab4aab1f8952dbc78610314f71a76"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.644773 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal/instance:
uuid: "674345a3abc24d99be9bf1909d4d0978"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "a21d0c6b9d2d5f8816d2f50011071ba5"
server_key_iv: "92bab4aab1f8952dbc78610314f71a76"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.646245 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:06:55.647011 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:55.647245 13769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:06:55.647362 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal
uuid: "674345a3abc24d99be9bf1909d4d0978"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "a21d0c6b9d2d5f8816d2f50011071ba5"
server_key_iv: "92bab4aab1f8952dbc78610314f71a76"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.647440 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:55.654856 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } attempt: 2
I20260928 09:06:55.655222 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.655102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33378)
0928 09:06:55.655156 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.655187 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:06:55.655519 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.655425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33382)
0928 09:06:55.655467 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.655490 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 09:06:55.655603 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40565: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40565: connect: Connection refused (error 111)
I20260928 09:06:55.655757 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:55.658032 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:55.658196 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:55.662400 13769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40565
I20260928 09:06:55.662446 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40565 every 8 connection(s)
I20260928 09:06:55.662817 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:06:55.663642 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:55.665136 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:55.671286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13769
I20260928 09:06:55.671389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-1/wal/instance
I20260928 09:06:55.671618 392 external_mini_cluster.cc:1428] Setting key 88372641b70775a23cf8df2a3b2d318f
I20260928 09:06:55.674106 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.665868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.98.62:38371)
0928 09:06:55.665926 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.665928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.666149 (+ 221us) server_negotiation.cc:407] Connection header received
0928 09:06:55.666594 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.666595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.666603 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.666666 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.667248 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.667744 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.668680 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.668940 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.668995 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.671062 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.671063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.671063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.671069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.672536 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.673056 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.673057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.673058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.673127 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.673387 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.673388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.673388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.673562 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.673628 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.673928 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.674016 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:06:55.674372 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.665855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38371 (local address 127.0.0.1:35040)
0928 09:06:55.666018 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.666058 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.666513 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.666717 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.666720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.666979 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.667171 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.667175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.667823 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.667823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.668589 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.668591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.669020 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.669020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.669126 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.669678 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.669686 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.670980 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.672626 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.672635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.672637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.672965 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.673214 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.673214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.673215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.673319 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.673653 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.673660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.673789 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.673882 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.674098 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260928 09:06:55.674926 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 09:06:55.683051 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.675201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33396)
0928 09:06:55.675423 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.675434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.675487 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.675649 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.675650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.675801 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.675861 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.675863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.676750 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.676751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.677659 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.677662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.678102 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.678103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.678182 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.678587 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.678592 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.680010 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.681995 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.681996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.681997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.682167 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.682388 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.682388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.682389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.682442 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.682741 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.682742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.682779 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.682840 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.682938 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260928 09:06:55.683213 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.675277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.0.98.61:40565)
0928 09:06:55.675415 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.675419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.675481 (+ 62us) server_negotiation.cc:407] Connection header received
0928 09:06:55.675546 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.675548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.675558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.675621 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.675938 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.676634 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.677737 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.678021 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.678078 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.680119 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.680128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.680128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.680136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.681897 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.682262 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.682265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.682266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.682324 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.682514 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.682515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.682515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.682664 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.682751 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.682867 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.682989 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
I20260928 09:06:55.684703 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:06:55.685055 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.684936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48212)
0928 09:06:55.684983 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.685030 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:06:55.685191 13786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
I20260928 09:06:55.685616 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.685502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48218)
0928 09:06:55.685571 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.685594 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:06:55.685725 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 09:06:55.701021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43065
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38371,127.0.98.61:40565,127.0.98.60:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:06:55.734941 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 1
I20260928 09:06:55.735426 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.735300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48232)
0928 09:06:55.735363 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.735398 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 09:06:55.735785 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.735690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48240)
0928 09:06:55.735735 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.735760 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 09:06:55.735872 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
I20260928 09:06:55.746737 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } attempt: 3
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:06:55.754331 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.747089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33410)
0928 09:06:55.747132 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.747151 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.747219 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.747365 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.747366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.747535 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.747602 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.747604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.748330 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.748331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.748864 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.748866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.749252 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.749254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.749337 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.749887 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.749893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.751351 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.753170 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.753171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.753172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.753359 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.753631 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.753633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.753633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.753701 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.753983 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.753984 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.754034 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.754120 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.754230 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 09:06:55.754331 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.747162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.98.61:40565)
0928 09:06:55.747232 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.747234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.747247 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:06:55.747278 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.747279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.747291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.747361 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.747698 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.748228 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.748956 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.749175 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.749255 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.751458 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.751460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.751460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.751468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.753047 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.753463 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.753464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.753464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.753525 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.753775 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.753776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.753776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.753895 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.753988 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.754157 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.754237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:06:55.754731 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:06:55.755085 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.754980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48254)
0928 09:06:55.755032 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.755059 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 09:06:55.755409 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.755311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48262)
0928 09:06:55.755363 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.755386 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 09:06:55.755540 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
W20260928 09:06:55.783455 13844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:55.783656 13844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:55.783679 13844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:55.784821 13844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:55.784864 13844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:55.784878 13844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:55.784891 13844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:55.784904 13844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:55.786517 13844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38371,127.0.98.61:40565,127.0.98.60:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:55.786693 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 1
I20260928 09:06:55.786803 13844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:55.787082 13844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:06:55.787192 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.787039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48266)
0928 09:06:55.787103 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.787139 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:06:55.787602 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.787508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48272)
0928 09:06:55.787553 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.787581 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:06:55.787742 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
W20260928 09:06:55.792169 13850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:55.792165 13849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:55.792245 13852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:55.792449 13844 server_base.cc:1034] running on GCE node
I20260928 09:06:55.792738 13844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:55.793012 13844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:55.794181 13844 hybrid_clock.cc:648] HybridClock initialized: now 1790586415794156 us; error 35 us; skew 500 ppm
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:55.796653 13844 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 09:06:55.797111 13844 webserver.cc:460] Webserver started at http://127.0.98.60:45925/ using document root <none> and password file <none>
I20260928 09:06:55.797345 13844 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:55.797415 13844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:55.797619 13844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:06:55.798681 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data/instance:
uuid: "43c48f6818b544bea7d82d04ae617f11"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "767ca4fb2e42e842f65cce13fd55ba73"
server_key_iv: "5595b4c45afbcac69380d7bcabd3a1da"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.799041 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal/instance:
uuid: "43c48f6818b544bea7d82d04ae617f11"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "767ca4fb2e42e842f65cce13fd55ba73"
server_key_iv: "5595b4c45afbcac69380d7bcabd3a1da"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.800508 13844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:06:55.801241 13859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:55.801409 13844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:06:55.801508 13844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal
uuid: "43c48f6818b544bea7d82d04ae617f11"
format_stamp: "Formatted at 2026-09-28 09:06:55 on dist-test-slave-smns"
server_key: "767ca4fb2e42e842f65cce13fd55ba73"
server_key_iv: "5595b4c45afbcac69380d7bcabd3a1da"
server_key_version: "encryptionkey@0"
I20260928 09:06:55.801666 13844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:55.804013 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 2
I20260928 09:06:55.804399 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.804259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48274)
0928 09:06:55.804323 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.804362 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:06:55.804792 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.804661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48286)
0928 09:06:55.804728 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.804760 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 09:06:55.804900 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
I20260928 09:06:55.819059 13844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:55.821403 13844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:55.821645 13844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:55.825742 13844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43065
I20260928 09:06:55.825804 13911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43065 every 8 connection(s)
I20260928 09:06:55.826172 13844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:06:55.827031 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:55.828586 13912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:55.835638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13844
I20260928 09:06:55.835770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790586261566264-392-0/minicluster-data/master-2/wal/instance
I20260928 09:06:55.835961 392 external_mini_cluster.cc:1428] Setting key 5c568ed10468c268dc76e439d77f9059
I20260928 09:06:55.837430 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.829305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.98.62:38371)
0928 09:06:55.829383 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.829385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.829644 (+ 259us) server_negotiation.cc:407] Connection header received
0928 09:06:55.830103 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.830104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.830141 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.830195 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.830799 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.831277 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.832220 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.832395 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.832446 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.834519 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.834520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.834520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.834527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.835853 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.836302 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.836303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.836303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.836361 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.836685 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.836687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.836687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.836846 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.836916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.837297 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.837366 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:06:55.837699 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.829283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38371 (local address 127.0.0.1:35044)
0928 09:06:55.829473 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.829509 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.830016 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.830267 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.830270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.830521 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.830728 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.830731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.831352 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.831353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.832146 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.832148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.832466 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.832466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.832573 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.833024 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.833033 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.834410 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.835928 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.835934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.835936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.836221 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.836436 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.836437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.836437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.836587 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.836926 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.836934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.837105 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.837272 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.837443 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260928 09:06:55.838266 13912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:06:55.845368 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.836339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.98.60:43065)
0928 09:06:55.836514 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.836518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.836578 (+ 60us) server_negotiation.cc:407] Connection header received
0928 09:06:55.836664 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.836666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.836681 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.836763 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.837494 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.838505 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.839473 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.839815 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.839884 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.842304 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.842312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.842312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.842318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.843971 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.844385 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.844388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.844388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.844489 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.844678 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.844679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.844679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.844881 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.845035 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.845136 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.845236 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260928 09:06:55.845496 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.838530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40565 (local address 127.0.0.1:33418)
0928 09:06:55.838573 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.838581 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.838614 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.838782 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.838783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.838891 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.838933 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.838934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.839717 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.839719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.840267 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.840269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.840642 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.840643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.840718 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.841098 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.841102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.842656 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.844438 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.844439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.844439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.844672 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.844855 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.844856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.844856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.844918 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.845185 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.845185 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.845224 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.845298 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.845373 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 09:06:55.845571 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.838621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.0.98.61:40565)
0928 09:06:55.838667 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.838669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.838676 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:06:55.838701 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.838702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.838711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.838771 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.839006 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.839650 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.840369 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.840560 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.840642 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.842733 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.842735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.842735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.842743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.844368 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.844748 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.844749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.844749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.844793 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.844980 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.844981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.844981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.845117 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.845187 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.845339 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.845411 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:06:55.845897 13912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:55.852469 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.846207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48300)
0928 09:06:55.846239 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.846248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.846287 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.846392 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.846393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.846522 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.846546 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.846547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.847200 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.847201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.847661 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.847662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.848042 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.848043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.848090 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.848419 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.848423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.849698 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.851378 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.851379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.851379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.851605 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.851802 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.851803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.851803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.851865 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.852144 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.852145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.852191 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.852274 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.852365 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 09:06:55.852484 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.846232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.0.98.60:43065)
0928 09:06:55.846264 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.846266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.846275 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:06:55.846340 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.846341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.846349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.846391 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.846624 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.847101 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.847756 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.847960 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.848043 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.849778 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.849779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.849780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.849786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.851270 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.851688 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.851689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.851689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.851738 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.851935 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.851936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.851936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.852069 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.852139 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.852291 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.852406 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":17}
I20260928 09:06:55.854146 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.846428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.98.62:38371)
0928 09:06:55.846470 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.846472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.846479 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:06:55.846580 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.846581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.846590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.846645 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.847120 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.847574 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.848472 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.848676 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.848762 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.851051 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.851053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.851053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.851061 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.852773 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.853096 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.853097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.853097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.853153 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.853350 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.853351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.853351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.853448 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.853971 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.854003 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.854064 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:06:55.854260 13912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: Bootstrap starting.
I20260928 09:06:55.855346 13912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:55.855708 13912 log.cc:824] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:55.856531 13912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: No bootstrap required, opened a new log
I20260928 09:06:55.858848 13912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.858983 13912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:55.859007 13912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c48f6818b544bea7d82d04ae617f11, State: Initialized, Role: FOLLOWER
I20260928 09:06:55.859174 13912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:06:55.859494 13912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:55.859470 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.859637 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:06:55.861372 13929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:06:55.861455 13929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:06:55.862870 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.854833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.61:40565)
0928 09:06:55.854869 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.854871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.854883 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:06:55.854958 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.854959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.854966 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.855010 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.855577 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.856165 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.857032 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.857219 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.857272 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.859877 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.859878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.859879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.859886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.861667 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.862041 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.862042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.862042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.862103 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.862313 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.862313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.862314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.862446 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.862601 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.862695 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.862772 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:06:55.862888 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 2
I20260928 09:06:55.861500 13844 master_runner.cc:186] Error getting master registration for 127.0.98.62:38371: OK, instance_id { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" instance_seqno: 1790586415516444 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:06:55.863916 13844 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:06:55.864490 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:06:55.870713 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.863104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48308)
0928 09:06:55.863250 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.863262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.863317 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.863479 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.863480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.863632 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.863671 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.863673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.864553 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.864554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.865328 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.865332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.865884 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.865886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.865949 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.866353 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.866358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.867785 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.869571 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.869572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.869572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.869781 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.870016 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.870017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.870017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.870082 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.870411 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.870412 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.870454 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.870518 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.870607 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 09:06:55.871064 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.863197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.98.60:43065)
0928 09:06:55.863233 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.863235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.863320 (+ 85us) server_negotiation.cc:407] Connection header received
0928 09:06:55.863380 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.863381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.863390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.863434 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.863756 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.864456 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.865425 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.865761 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.865832 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.867877 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.867878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.867879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.867885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.869466 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.869861 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.869862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.869862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.869926 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.870163 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.870164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.870164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.870323 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.870385 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.870899 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.870973 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:06:55.872825 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: Bootstrap starting.
I20260928 09:06:55.873572 13762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:55.873582 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.865881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.98.61:40565)
0928 09:06:55.865921 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.865923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.866222 (+ 299us) server_negotiation.cc:407] Connection header received
0928 09:06:55.866909 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.866911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.866921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.866976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.867584 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.868014 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.868918 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.869118 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.869218 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.870581 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.870582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.870583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.870589 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.872263 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.872658 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.872659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.872660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.872719 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.872948 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.872949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.872949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.873070 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.873158 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.873377 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.873455 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:06:55.873893 13762 log.cc:824] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:55.878279 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: No bootstrap required, opened a new log
I20260928 09:06:55.878868 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.866369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.98.60:43065)
0928 09:06:55.871385 (+ 5016us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.871386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.871393 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:06:55.871410 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.871411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.871420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.871474 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.872095 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.872716 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.873695 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.873966 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.874008 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.875282 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.875283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.875283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.875289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.877038 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.877404 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.877405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.877405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.877457 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.877760 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.877762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.877762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.877896 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.878238 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.878469 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.878776 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5004,"thread_start_us":83,"threads_started":1}
I20260928 09:06:55.880175 13762 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.880273 13762 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:55.880296 13762 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d5e302e5cc4102866483b8721d9082, State: Initialized, Role: FOLLOWER
I20260928 09:06:55.880420 13762 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.880498 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.866250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.98.62:38371)
0928 09:06:55.866287 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.866289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.873697 (+ 7408us) server_negotiation.cc:407] Connection header received
0928 09:06:55.873792 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.873793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.873803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.873834 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.874537 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.875226 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.876172 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.876381 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.876435 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.877776 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.877777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.877777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.877784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.879289 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.879662 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.879663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.879663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.879712 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.879893 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.879893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.879894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.880017 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.880078 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.880301 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.880374 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:06:55.880753 13762 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:55.880769 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.880857 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:06:55.882567 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:06:55.882628 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:06:55.883368 13694 master_runner.cc:186] Error getting master registration for 127.0.98.61:40565: OK, instance_id { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" instance_seqno: 1790586415661077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:06:55.885936 13694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:06:55.896030 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 3
Sep 28 09:06:55 dist-test-slave-smns krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:06:55.903218 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.896553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.98.60:43065)
0928 09:06:55.896606 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:55.896608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:55.896614 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:06:55.896635 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:55.896636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:55.896646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:55.896694 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:55.897080 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.897635 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.898320 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.898523 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.898600 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.900531 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:55.900533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:55.900533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:55.900541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:55.902028 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.902432 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.902434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.902434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.902495 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:55.902695 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:55.902696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:55.902697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:55.902814 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:55.902887 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:55.903044 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 09:06:55.903121 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:06:55.903226 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:55.896337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:48334)
0928 09:06:55.896427 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:55.896440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:55.896481 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:55.896745 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:55.896748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:55.896946 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:55.896996 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.896998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.897723 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.897724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.898229 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:55.898231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:55.898596 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:55.898597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:55.898662 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:55.899118 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:55.899123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:55.900437 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:55.902162 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.902163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.902164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.902356 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.902565 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:55.902566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:55.902566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:55.902624 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:55.902891 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:55.902892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:55.902938 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 09:06:55.903021 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:06:55.903121 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 09:06:55.904946 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978: Bootstrap starting.
I20260928 09:06:55.905735 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:55.906059 13837 log.cc:824] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:55.906869 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978: No bootstrap required, opened a new log
I20260928 09:06:55.908329 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.908469 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:55.908535 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674345a3abc24d99be9bf1909d4d0978, State: Initialized, Role: FOLLOWER
I20260928 09:06:55.908669 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.908979 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.909049 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:55.909116 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:06:55.910987 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:06:55.911098 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:06:55.913311 13769 master_runner.cc:263] No leader master found from master 674345a3abc24d99be9bf1909d4d0978
I20260928 09:06:55.913355 13769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:06:55.977356 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:06:55.977471 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.977731 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8d5e302e5cc4102866483b8721d9082 (127.0.98.62:38371), 43c48f6818b544bea7d82d04ae617f11 (127.0.98.60:43065)
I20260928 09:06:55.978241 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674345a3abc24d99be9bf1909d4d0978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c48f6818b544bea7d82d04ae617f11" is_pre_election: true
I20260928 09:06:55.978400 13886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674345a3abc24d99be9bf1909d4d0978 in term 0.
I20260928 09:06:55.978740 13786 leader_election.cc:302] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c48f6818b544bea7d82d04ae617f11, 674345a3abc24d99be9bf1909d4d0978; no voters:
I20260928 09:06:55.978969 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:06:55.979027 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:06:55.979053 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:55.980000 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.980121 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [CANDIDATE]: Term 1 election: Requested vote from peers b8d5e302e5cc4102866483b8721d9082 (127.0.98.62:38371), 43c48f6818b544bea7d82d04ae617f11 (127.0.98.60:43065)
I20260928 09:06:55.980501 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674345a3abc24d99be9bf1909d4d0978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c48f6818b544bea7d82d04ae617f11"
I20260928 09:06:55.980594 13886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:55.981483 13886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674345a3abc24d99be9bf1909d4d0978 in term 1.
I20260928 09:06:55.981818 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674345a3abc24d99be9bf1909d4d0978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d5e302e5cc4102866483b8721d9082"
I20260928 09:06:55.981950 13735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:55.982772 13735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674345a3abc24d99be9bf1909d4d0978 in term 1.
I20260928 09:06:55.982926 13736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674345a3abc24d99be9bf1909d4d0978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d5e302e5cc4102866483b8721d9082" is_pre_election: true
I20260928 09:06:55.983026 13736 raft_consensus.cc:2362] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 674345a3abc24d99be9bf1909d4d0978 in term 1. Re-sending same reply.
I20260928 09:06:55.983285 13788 leader_election.cc:302] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674345a3abc24d99be9bf1909d4d0978, b8d5e302e5cc4102866483b8721d9082; no voters:
I20260928 09:06:55.983494 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:55.983665 13954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [term 1 LEADER]: Becoming Leader. State: Replica: 674345a3abc24d99be9bf1909d4d0978, State: Running, Role: LEADER
I20260928 09:06:55.983747 13954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260928 09:06:55.984114 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674345a3abc24d99be9bf1909d4d0978. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.984184 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:55.985560 13969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:06:55.985801 13969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:06:55.987888 13886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [term 1 FOLLOWER]: Refusing update from remote peer 674345a3abc24d99be9bf1909d4d0978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:06:55.988242 13954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:06:55.989630 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674345a3abc24d99be9bf1909d4d0978. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.989753 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:06:55.990211 13736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [term 1 FOLLOWER]: Refusing update from remote peer 674345a3abc24d99be9bf1909d4d0978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:06:55.990550 13954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:06:55.991729 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674345a3abc24d99be9bf1909d4d0978. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.991847 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:06:55.993360 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.993456 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:55.993935 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.994016 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:06:55.994362 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.994438 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978 [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:55.994751 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "674345a3abc24d99be9bf1909d4d0978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d5e302e5cc4102866483b8721d9082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38371 } } peers { permanent_uuid: "674345a3abc24d99be9bf1909d4d0978" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40565 } } peers { permanent_uuid: "43c48f6818b544bea7d82d04ae617f11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260928 09:06:55.994827 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:06:55.995210 13970 mvcc.cc:205] Tried to move back new op lower bound from 7334241959883882496 to 7334241959869972480. Current Snapshot: MvccSnapshot[applied={T|T < 7334241959883882496}]
I20260928 09:06:55.995599 13973 mvcc.cc:205] Tried to move back new op lower bound from 7334241959883882496 to 7334241959869972480. Current Snapshot: MvccSnapshot[applied={T|T < 7334241959883882496}]
I20260928 09:06:55.996150 13971 mvcc.cc:205] Tried to move back new op lower bound from 7334241959883882496 to 7334241959869972480. Current Snapshot: MvccSnapshot[applied={T|T < 7334241959883882496}]
I20260928 09:06:55.996698 13969 catalog_manager.cc:1347] Generated new cluster ID: 0fde6c3b243d4a4f84d90c605108173d
I20260928 09:06:55.996743 13969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:06:56.024559 13969 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:06:56.025391 13969 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:06:56.046792 13969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 674345a3abc24d99be9bf1909d4d0978: Generated new TSK 0
I20260928 09:06:56.047120 13969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:06:56.861914 13929 catalog_manager.cc:1259] Loaded cluster ID: 0fde6c3b243d4a4f84d90c605108173d
I20260928 09:06:56.862016 13929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: loading cluster ID for follower catalog manager: success
I20260928 09:06:56.862859 13929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: acquiring CA information for follower catalog manager: success
I20260928 09:06:56.863256 13929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43c48f6818b544bea7d82d04ae617f11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:06:56.883044 13951 catalog_manager.cc:1259] Loaded cluster ID: 0fde6c3b243d4a4f84d90c605108173d
I20260928 09:06:56.883124 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: loading cluster ID for follower catalog manager: success
I20260928 09:06:56.883950 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: acquiring CA information for follower catalog manager: success
I20260928 09:06:56.884305 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8d5e302e5cc4102866483b8721d9082: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:06:57.105280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13694
I20260928 09:06:57.110942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13769
I20260928 09:06:57.115757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13844
2026-09-28T09:06:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2913 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:06:57 dist-test-slave-smns krb5kdc[13983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:06:57 dist-test-slave-smns krb5kdc[13983](info): set up 2 sockets
Sep 28 09:06:57 dist-test-slave-smns krb5kdc[13983](info): commencing operation
krb5kdc: starting...
W20260928 09:06:58.149813 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:06:58 dist-test-slave-smns krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:06:58.269418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:58.355449 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:58.355719 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:58.355765 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:58.356948 13996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:58.357016 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:58.357041 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:58.357071 13996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:58.357154 13996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:58.358690 13996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:58.358968 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:58.359200 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:58.364058 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:58.364078 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:58.364166 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:58.364243 13996 server_base.cc:1034] running on GCE node
I20260928 09:06:58.364583 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:58.364916 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:58.366075 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790586418366060 us; error 43 us; skew 500 ppm
Sep 28 09:06:58 dist-test-slave-smns krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:58.368574 13996 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:06:58.369040 13996 webserver.cc:460] Webserver started at http://127.0.98.62:40401/ using document root <none> and password file <none>
I20260928 09:06:58.369292 13996 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:58.369362 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:58.369509 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:06:58.370585 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "2ebb81dc5a9f4de2861effd49489feec"
format_stamp: "Formatted at 2026-09-28 09:06:58 on dist-test-slave-smns"
server_key: "09142ae56787653c079e828fd72e09c7"
server_key_iv: "6048a9ffa5fdf2ca20a28092a875bd02"
server_key_version: "encryptionkey@0"
I20260928 09:06:58.370944 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ebb81dc5a9f4de2861effd49489feec"
format_stamp: "Formatted at 2026-09-28 09:06:58 on dist-test-slave-smns"
server_key: "09142ae56787653c079e828fd72e09c7"
server_key_iv: "6048a9ffa5fdf2ca20a28092a875bd02"
server_key_version: "encryptionkey@0"
I20260928 09:06:58.372247 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:06:58.372946 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:58.373117 13996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:06:58.373220 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "2ebb81dc5a9f4de2861effd49489feec"
format_stamp: "Formatted at 2026-09-28 09:06:58 on dist-test-slave-smns"
server_key: "09142ae56787653c079e828fd72e09c7"
server_key_iv: "6048a9ffa5fdf2ca20a28092a875bd02"
server_key_version: "encryptionkey@0"
I20260928 09:06:58.373293 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:58.413215 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:58.415235 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:58.415405 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:58.419502 13996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41709
I20260928 09:06:58.419540 14063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41709 every 8 connection(s)
I20260928 09:06:58.419904 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:06:58.420852 14064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:58.423059 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec: Bootstrap starting.
I20260928 09:06:58.423604 14064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:58.423866 14064 log.cc:824] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:58.424492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 13996
I20260928 09:06:58.424499 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec: No bootstrap required, opened a new log
I20260928 09:06:58.424580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:06:58.424795 392 external_mini_cluster.cc:1428] Setting key 233e00cf4dad4f162db4a8a5fd0423ed
I20260928 09:06:58.426868 14064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebb81dc5a9f4de2861effd49489feec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260928 09:06:58.427081 14064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:58.427124 14064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebb81dc5a9f4de2861effd49489feec, State: Initialized, Role: FOLLOWER
I20260928 09:06:58.427279 14064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebb81dc5a9f4de2861effd49489feec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260928 09:06:58.427361 14064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:06:58.427390 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:06:58.427434 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:06:58.428125 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebb81dc5a9f4de2861effd49489feec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260928 09:06:58.428228 14064 leader_election.cc:302] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebb81dc5a9f4de2861effd49489feec; no voters:
I20260928 09:06:58.428375 14064 leader_election.cc:288] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:06:58.428426 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:58.428601 14064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:58.428615 14069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [term 1 LEADER]: Becoming Leader. State: Replica: 2ebb81dc5a9f4de2861effd49489feec, State: Running, Role: LEADER
I20260928 09:06:58.428821 14069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebb81dc5a9f4de2861effd49489feec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260928 09:06:58.429774 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ebb81dc5a9f4de2861effd49489feec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebb81dc5a9f4de2861effd49489feec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260928 09:06:58.429867 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:58.430061 14082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:06:58.430305 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebb81dc5a9f4de2861effd49489feec. Latest consensus state: current_term: 1 leader_uuid: "2ebb81dc5a9f4de2861effd49489feec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebb81dc5a9f4de2861effd49489feec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260928 09:06:58.430492 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:58.430696 14082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:06:58 dist-test-slave-smns krb5kdc[13983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:58.432905 14082 catalog_manager.cc:1347] Generated new cluster ID: 8e45fd7331ba42459e232460e54be0b0
I20260928 09:06:58.432960 14082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:06:58.437152 14082 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:06:58.437179 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:58.425276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.0.98.62:41709)
0928 09:06:58.425472 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:58.425482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:58.425515 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:06:58.426060 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:58.426069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:58.426131 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:58.426284 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:58.426874 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:58.428603 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:58.429826 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:58.430483 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:58.430593 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:58.433249 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:58.433253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:58.433253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:58.433265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:58.435344 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:58.435745 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:58.435748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:58.435748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:58.435870 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:58.436086 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:58.436087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:58.436088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:58.436405 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:58.436582 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:58.436671 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:06:58.436808 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260928 09:06:58.438122 14082 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:06:58.438405 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:06:58.447959 14082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ebb81dc5a9f4de2861effd49489feec: Generated new TSK 0
I20260928 09:06:58.448156 14082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:06:58.492312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 13996
2026-09-28T09:06:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1374 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:06:58 dist-test-slave-smns krb5kdc[14091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:06:58 dist-test-slave-smns krb5kdc[14091](info): set up 2 sockets
Sep 28 09:06:58 dist-test-slave-smns krb5kdc[14091](info): commencing operation
krb5kdc: starting...
W20260928 09:06:59.529789 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:06:59 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:06:59.657006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:59.738535 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:59.738801 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:59.738839 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:59.740015 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:06:59.740087 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:59.740119 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:59.740147 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:06:59.740180 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:06:59.741710 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:59.741999 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:59.742231 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:59.747118 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:59.747200 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:59.747217 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:59.747471 14104 server_base.cc:1034] running on GCE node
I20260928 09:06:59.747757 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:59.748055 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:59.749250 14104 hybrid_clock.cc:648] HybridClock initialized: now 1790586419749217 us; error 48 us; skew 500 ppm
Sep 28 09:06:59 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:59.752024 14104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:06:59.752557 14104 webserver.cc:460] Webserver started at http://127.0.98.62:37773/ using document root <none> and password file <none>
I20260928 09:06:59.752807 14104 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:59.752875 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:59.753011 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:06:59.754190 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "4865aae91b704fe5ab0a862347575e1d"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "3a1b2399f4bea1dc94e5ef8c73f6c2fa"
server_key_iv: "d661f16cb5bcbf35c54c02dd8d55ccd4"
server_key_version: "encryptionkey@0"
I20260928 09:06:59.754581 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "4865aae91b704fe5ab0a862347575e1d"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "3a1b2399f4bea1dc94e5ef8c73f6c2fa"
server_key_iv: "d661f16cb5bcbf35c54c02dd8d55ccd4"
server_key_version: "encryptionkey@0"
I20260928 09:06:59.755957 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:06:59.756826 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:59.757033 14104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:06:59.757149 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "4865aae91b704fe5ab0a862347575e1d"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "3a1b2399f4bea1dc94e5ef8c73f6c2fa"
server_key_iv: "d661f16cb5bcbf35c54c02dd8d55ccd4"
server_key_version: "encryptionkey@0"
I20260928 09:06:59.757231 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:59.773878 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:59.775812 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:59.775976 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:59.780004 14104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44509
I20260928 09:06:59.780037 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44509 every 8 connection(s)
I20260928 09:06:59.780400 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:06:59.781363 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:06:59.782123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14104
I20260928 09:06:59.782239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:06:59.782442 392 external_mini_cluster.cc:1428] Setting key 103109b3de948bf6becfc5a659dce8d0
I20260928 09:06:59.784698 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap starting.
I20260928 09:06:59.785311 14172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Neither blocks nor log segments found. Creating new log.
I20260928 09:06:59.785671 14172 log.cc:824] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:06:59.786422 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: No bootstrap required, opened a new log
I20260928 09:06:59.788880 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:06:59.789075 14172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:06:59.789127 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4865aae91b704fe5ab0a862347575e1d, State: Initialized, Role: FOLLOWER
I20260928 09:06:59.789260 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:06:59.789335 14172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:06:59.789357 14172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:06:59.789428 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 0 FOLLOWER]: Advancing to term 1
Sep 28 09:06:59 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:59.790092 14172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:06:59.790215 14172 leader_election.cc:302] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4865aae91b704fe5ab0a862347575e1d; no voters:
I20260928 09:06:59.790392 14172 leader_election.cc:288] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:06:59.790472 14177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:06:59.790669 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:06:59.790678 14177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 LEADER]: Becoming Leader. State: Replica: 4865aae91b704fe5ab0a862347575e1d, State: Running, Role: LEADER
I20260928 09:06:59.790902 14177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:06:59.791338 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4865aae91b704fe5ab0a862347575e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } } }
I20260928 09:06:59.791337 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4865aae91b704fe5ab0a862347575e1d. Latest consensus state: current_term: 1 leader_uuid: "4865aae91b704fe5ab0a862347575e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } } }
I20260928 09:06:59.791473 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:59.791481 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: This master's current role is: LEADER
I20260928 09:06:59.791972 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:06:59.792763 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:06:59.794729 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:59.783029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.0.98.62:44509)
0928 09:06:59.783239 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:59.783247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:59.783272 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:06:59.783672 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:59.783678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:59.783724 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:59.783841 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:59.784680 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:59.785805 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:59.786821 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:59.787314 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:59.787405 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:59.790348 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:59.790352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:59.790353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:59.790365 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:59.792814 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:59.793276 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:59.793279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:59.793280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:59.793382 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:59.793663 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:59.793664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:59.793664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:59.793957 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:59.794109 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:59.794376 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 09:06:59.794521 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260928 09:06:59.794862 14186 catalog_manager.cc:1347] Generated new cluster ID: 92846394d05e4c26bf752c2de15982cf
I20260928 09:06:59.794898 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:06:59.815374 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:06:59.815945 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:06:59.820760 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Generated new TSK 0
I20260928 09:06:59.820940 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:06:59.853120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44509
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:06:59.934939 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:06:59.935151 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:06:59.935175 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:06:59.936348 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:06:59.936390 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:06:59.936417 14198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:06:59.937893 14198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:06:59.938138 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:06:59.938385 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:06:59.943398 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:59.943401 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:06:59.943406 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:06:59.943660 14198 server_base.cc:1034] running on GCE node
I20260928 09:06:59.943864 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:06:59.944089 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:06:59.945225 14198 hybrid_clock.cc:648] HybridClock initialized: now 1790586419945207 us; error 40 us; skew 500 ppm
Sep 28 09:06:59 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:06:59.947804 14198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:06:59.948304 14198 webserver.cc:460] Webserver started at http://127.0.98.1:44339/ using document root <none> and password file <none>
I20260928 09:06:59.948532 14198 fs_manager.cc:359] Metadata directory not provided
I20260928 09:06:59.948611 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:06:59.948764 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:06:59.949885 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9ed33ef255548b2bc91ef32cb5a0686"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "d7ccdfe92d26f2361eef2e7945f2d5a7"
server_key_iv: "679a35c479305649d27bcf903dbf3010"
server_key_version: "encryptionkey@0"
I20260928 09:06:59.950254 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9ed33ef255548b2bc91ef32cb5a0686"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "d7ccdfe92d26f2361eef2e7945f2d5a7"
server_key_iv: "679a35c479305649d27bcf903dbf3010"
server_key_version: "encryptionkey@0"
I20260928 09:06:59.951608 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:06:59.952364 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:59.952548 14198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:06:59.952672 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "d9ed33ef255548b2bc91ef32cb5a0686"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "d7ccdfe92d26f2361eef2e7945f2d5a7"
server_key_iv: "679a35c479305649d27bcf903dbf3010"
server_key_version: "encryptionkey@0"
I20260928 09:06:59.952751 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:06:59.975313 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:06:59.977210 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:06:59.977377 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:06:59.977672 14198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:06:59.978001 14198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:06:59.978056 14198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:59.978089 14198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:06:59.978107 14198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:06:59.984123 14198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38043
I20260928 09:06:59.984176 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38043 every 8 connection(s)
I20260928 09:06:59.984491 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:06:59.988510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14198
I20260928 09:06:59.988620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:06:59.988816 392 external_mini_cluster.cc:1428] Setting key fde6f5c3070cd81c34c504536fd8ff8d
Sep 28 09:06:59 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:06:59.995930 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:59.985298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34191 (local address 127.0.98.62:44509)
0928 09:06:59.985380 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 09:06:59.985383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:06:59.985534 (+ 151us) server_negotiation.cc:407] Connection header received
0928 09:06:59.986200 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:06:59.986201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:06:59.986210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:06:59.986260 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:06:59.987295 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:59.987801 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:59.989292 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:06:59.989596 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:06:59.989658 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:59.992545 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:06:59.992547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:06:59.992547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:06:59.992557 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:06:59.994478 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:59.994917 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:59.994918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:59.994918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:59.994986 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:06:59.995240 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:06:59.995242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:06:59.995242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:06:59.995398 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:06:59.995473 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:06:59.995730 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 09:06:59.995825 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 09:06:59.996007 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:06:59.985205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:34191)
0928 09:06:59.985365 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 09:06:59.985436 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 09:06:59.986091 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:06:59.986337 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:06:59.986340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:06:59.986969 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:06:59.987189 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:59.987194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:59.987891 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:59.987892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:59.989199 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:06:59.989202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:06:59.989680 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:06:59.989682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:06:59.989819 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:06:59.990490 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:06:59.990499 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:06:59.992404 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:06:59.994578 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:59.994585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:59.994587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:59.994835 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:59.995063 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:06:59.995064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:06:59.995064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:06:59.995171 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:06:59.995472 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:06:59.995476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:06:59.995597 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 09:06:59.995680 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:06:59.995802 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260928 09:06:59.996543 14327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44509
I20260928 09:06:59.996658 14327 heartbeater.cc:464] Registering TS with master...
I20260928 09:06:59.996840 14327 heartbeater.cc:511] Master 127.0.98.62:44509 requested a full tablet report, sending...
I20260928 09:06:59.997532 14136 ts_manager.cc:194] Registered new tserver with Master: d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 09:06:59.998457 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:07:00.017462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44509
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:00.099751 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:00.100021 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:00.100061 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:00.101270 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:00.101348 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:00.101393 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:07:00.102963 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:00.103266 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:00.103516 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:00.108842 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.108830 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.108922 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.109300 14332 server_base.cc:1034] running on GCE node
I20260928 09:07:00.109529 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:00.109829 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:00.111002 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790586420110988 us; error 47 us; skew 500 ppm
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:00.113919 14332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:07:00.114403 14332 webserver.cc:460] Webserver started at http://127.0.98.2:36489/ using document root <none> and password file <none>
I20260928 09:07:00.114660 14332 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:00.114729 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:00.114882 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:07:00.116259 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "408654adb033416fa94b1337568c1049"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "b969a49444897ea692190fa126831244"
server_key_iv: "cd8a38bba62ab7191baecc9fc83e5f06"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.116724 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "408654adb033416fa94b1337568c1049"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "b969a49444897ea692190fa126831244"
server_key_iv: "cd8a38bba62ab7191baecc9fc83e5f06"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.118363 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:07:00.119235 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.119436 14332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:00.119544 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "408654adb033416fa94b1337568c1049"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "b969a49444897ea692190fa126831244"
server_key_iv: "cd8a38bba62ab7191baecc9fc83e5f06"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.119627 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:00.130322 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:00.132500 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:00.132665 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:00.132887 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:00.133209 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:00.133262 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.133304 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:00.133332 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.139570 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40525
I20260928 09:07:00.139622 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40525 every 8 connection(s)
I20260928 09:07:00.139968 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:07:00.142316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14332
I20260928 09:07:00.142395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:07:00.142587 392 external_mini_cluster.cc:1428] Setting key 93438ebe6ea3548cb833258b0ca9386e
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:07:00.151599 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.140737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59059 (local address 127.0.98.62:44509)
0928 09:07:00.140792 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.140795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.141004 (+ 209us) server_negotiation.cc:407] Connection header received
0928 09:07:00.141674 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.141676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.141689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.141820 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.143075 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.143754 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.145186 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.145379 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.145441 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.148191 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.148193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.148193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.148201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.150076 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.150564 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.150566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.150566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.150628 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.150897 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.150898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.150899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.151028 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.151105 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.151392 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.151492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 09:07:00.151822 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.140676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:59059)
0928 09:07:00.140854 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.140906 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.141508 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.141904 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.141908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.142674 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:00.142963 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.142970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.143866 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.143867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.145076 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.145088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.145474 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.145475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.145685 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.146276 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:00.146289 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:00.148077 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:00.150184 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.150193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.150195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.150444 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.150696 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.150697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.150698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.150818 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.151097 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:00.151104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:00.151257 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.151371 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.151555 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260928 09:07:00.152468 14461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44509
I20260928 09:07:00.152604 14461 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:00.152808 14461 heartbeater.cc:511] Master 127.0.98.62:44509 requested a full tablet report, sending...
I20260928 09:07:00.153182 14136 ts_manager.cc:194] Registered new tserver with Master: 408654adb033416fa94b1337568c1049 (127.0.98.2:40525)
I20260928 09:07:00.153609 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:07:00.172660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44509
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:00.254709 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:00.254915 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:00.254940 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:00.256260 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:00.256302 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:00.256330 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:07:00.257947 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:00.258194 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:00.258426 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:00.263614 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.263643 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.263638 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.263700 14466 server_base.cc:1034] running on GCE node
I20260928 09:07:00.264066 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:00.264288 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:00.265471 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790586420265454 us; error 28 us; skew 500 ppm
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:00.268041 14466 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:07:00.268504 14466 webserver.cc:460] Webserver started at http://127.0.98.3:38995/ using document root <none> and password file <none>
I20260928 09:07:00.268755 14466 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:00.268826 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:00.268976 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:07:00.270105 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "5290392e54994aed81218dedb492ae17"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "e20efd37483c3809d6feb7a97d53f398"
server_key_iv: "b6a1c7d83ef579777e47a2bfeb0ae9b3"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.270474 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5290392e54994aed81218dedb492ae17"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "e20efd37483c3809d6feb7a97d53f398"
server_key_iv: "b6a1c7d83ef579777e47a2bfeb0ae9b3"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.271802 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:07:00.272603 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.272831 14466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:00.272938 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "5290392e54994aed81218dedb492ae17"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "e20efd37483c3809d6feb7a97d53f398"
server_key_iv: "b6a1c7d83ef579777e47a2bfeb0ae9b3"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.273012 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:00.288437 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:00.290820 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:00.291142 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:00.291507 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:00.291877 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:00.291939 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.291983 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:00.292006 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.298622 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34895
I20260928 09:07:00.298731 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34895 every 8 connection(s)
I20260928 09:07:00.299057 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:00.307636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14466
I20260928 09:07:00.307737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:07:00.307937 392 external_mini_cluster.cc:1428] Setting key c824d71d62161223fcd49d835779d9b2
I20260928 09:07:00.309515 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.300003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34831 (local address 127.0.98.62:44509)
0928 09:07:00.300047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.300074 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.300304 (+ 230us) server_negotiation.cc:407] Connection header received
0928 09:07:00.300978 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.300980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.300993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.301042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.302109 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.302621 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.303669 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.303887 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.303965 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.306416 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.306418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.306418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.306426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.307950 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.308554 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.308555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.308555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.308646 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.308932 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.308933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.308933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.309072 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.309129 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.309383 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.309450 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:07:00.309820 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.299959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:34831)
0928 09:07:00.300120 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.300169 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.300824 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.301117 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.301121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.301791 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:00.301997 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.302003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.302720 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.302722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.303564 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.303567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.303972 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.303973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.304142 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.304727 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:00.304737 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:00.306329 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:00.308060 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.308069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.308071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.308468 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.308714 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.308715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.308715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.308860 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.309138 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:00.309143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:00.309267 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.309355 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.309502 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 09:07:00.310329 14595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44509
I20260928 09:07:00.310424 14595 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:00.310571 14595 heartbeater.cc:511] Master 127.0.98.62:44509 requested a full tablet report, sending...
I20260928 09:07:00.310938 14136 ts_manager.cc:194] Registered new tserver with Master: 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895)
I20260928 09:07:00.311314 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34831
I20260928 09:07:00.319919 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:07:00.327729 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.321461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.0.98.62:44509)
0928 09:07:00.321513 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.321515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.321657 (+ 142us) server_negotiation.cc:407] Connection header received
0928 09:07:00.321696 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.321697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.321706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.321757 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.322359 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.322878 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.323644 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.323822 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.323873 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.325137 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.325139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.325139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.325145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.326599 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.327021 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.327022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.327023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.327075 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.327250 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.327250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.327251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.327394 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.327468 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.327598 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.327664 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 09:07:00.329406 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:07:00.331161 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:07:00.337677 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.333348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34895 (local address 127.0.0.1:40378)
0928 09:07:00.333794 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.333802 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.333858 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.334054 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.334058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.334060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:00.334133 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.334134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.335156 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.335157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.336388 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.336390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.337294 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.337295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.337374 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.337387 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.337487 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.337575 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":2,"thread_start_us":127,"threads_started":1}
I20260928 09:07:00.337882 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.333563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.0.98.2:40525)
0928 09:07:00.333877 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.333881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.333891 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:07:00.333931 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.333933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.334058 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.334220 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:00.334512 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.335555 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.336698 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.337394 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.337482 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.337534 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.337741 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.337789 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260928 09:07:00.337880 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.333641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.0.98.1:38043)
0928 09:07:00.333972 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.333976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.333984 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:07:00.334038 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.334040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.334056 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.334143 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:00.334335 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.335462 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.336684 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.337284 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.337352 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.337408 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.337735 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.337787 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":73,"threads_started":1}
I20260928 09:07:00.338114 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.333515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.0.98.3:34895)
0928 09:07:00.333802 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.333807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.333856 (+ 49us) server_negotiation.cc:407] Connection header received
0928 09:07:00.333930 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.333932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.333949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.334032 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:00.334250 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.335049 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.336470 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.337205 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.337280 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.337337 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.337612 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.337668 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":144,"threads_started":1}
I20260928 09:07:00.338131 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.333344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40525 (local address 127.0.0.1:48648)
0928 09:07:00.333567 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.333614 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.333691 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.334305 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.334306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.334308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:00.334359 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.334361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.335669 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.335670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.336608 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.336611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.337498 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.337500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.337603 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.337608 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.337661 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.337701 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":61,"threads_started":1}
I20260928 09:07:00.338760 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 052f950b61c8493b85915bca16c62cb7 (DEFAULT_TABLE table=test-table [id=4e641915409b4794a93b341ff64c5a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:00.338757 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 052f950b61c8493b85915bca16c62cb7 (DEFAULT_TABLE table=test-table [id=4e641915409b4794a93b341ff64c5a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:00.339082 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052f950b61c8493b85915bca16c62cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:00.339087 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052f950b61c8493b85915bca16c62cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:00.341955 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.333348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38043 (local address 127.0.0.1:57718)
0928 09:07:00.333757 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.333769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.333827 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.334220 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.334220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.334222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:00.334260 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.334261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.335568 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.335569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.336601 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.336603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.337439 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.337440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.337509 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.337512 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.337640 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.337674 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260928 09:07:00.341950 14613 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Bootstrap starting.
I20260928 09:07:00.342183 14612 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Bootstrap starting.
I20260928 09:07:00.342648 14613 tablet_bootstrap.cc:654] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:00.342757 14612 tablet_bootstrap.cc:654] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:00.342712 14261 tablet_service.cc:1467] Processing CreateTablet for tablet 052f950b61c8493b85915bca16c62cb7 (DEFAULT_TABLE table=test-table [id=4e641915409b4794a93b341ff64c5a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:00.343103 14261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052f950b61c8493b85915bca16c62cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:00.343236 14613 log.cc:824] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:00.343274 14612 log.cc:824] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:00.344008 14613 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: No bootstrap required, opened a new log
I20260928 09:07:00.344017 14612 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: No bootstrap required, opened a new log
I20260928 09:07:00.344084 14613 ts_tablet_manager.cc:1397] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:07:00.344084 14612 ts_tablet_manager.cc:1397] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:07:00.346114 14616 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap starting.
I20260928 09:07:00.346118 14612 raft_consensus.cc:348] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.346122 14613 raft_consensus.cc:348] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.346316 14613 raft_consensus.cc:374] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:00.346315 14612 raft_consensus.cc:374] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:00.346366 14612 raft_consensus.cc:729] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5290392e54994aed81218dedb492ae17, State: Initialized, Role: FOLLOWER
I20260928 09:07:00.346366 14613 raft_consensus.cc:729] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408654adb033416fa94b1337568c1049, State: Initialized, Role: FOLLOWER
I20260928 09:07:00.346484 14612 consensus_queue.cc:260] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.346542 14613 consensus_queue.cc:260] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.346779 14612 ts_tablet_manager.cc:1428] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:07:00.346801 14613 ts_tablet_manager.cc:1428] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:07:00.346840 14461 heartbeater.cc:503] Master 127.0.98.62:44509 was elected leader, sending a full tablet report...
I20260928 09:07:00.346907 14595 heartbeater.cc:503] Master 127.0.98.62:44509 was elected leader, sending a full tablet report...
I20260928 09:07:00.347134 14616 tablet_bootstrap.cc:654] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:00.347745 14616 log.cc:824] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:00.348511 14616 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: No bootstrap required, opened a new log
I20260928 09:07:00.348573 14616 ts_tablet_manager.cc:1397] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:07:00.350222 14616 raft_consensus.cc:348] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.350378 14616 raft_consensus.cc:374] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:00.350400 14616 raft_consensus.cc:729] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed33ef255548b2bc91ef32cb5a0686, State: Initialized, Role: FOLLOWER
I20260928 09:07:00.350520 14616 consensus_queue.cc:260] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.350772 14616 ts_tablet_manager.cc:1428] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:07:00.350842 14327 heartbeater.cc:503] Master 127.0.98.62:44509 was elected leader, sending a full tablet report...
I20260928 09:07:00.369154 14620 raft_consensus.cc:482] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:00.369343 14620 raft_consensus.cc:504] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.369840 14620 leader_election.cc:288] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895), 408654adb033416fa94b1337568c1049 (127.0.98.2:40525)
I20260928 09:07:00.372978 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.370078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40525 (local address 127.0.98.1:42519)
0928 09:07:00.370265 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.370276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.370340 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.370500 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.370501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.370502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:00.370575 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.370577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.371220 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.371220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.371962 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.371964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.372697 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.372698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.372778 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.372782 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.372836 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.372879 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260928 09:07:00.372987 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.370197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42519 (local address 127.0.98.2:40525)
0928 09:07:00.370257 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.370259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.370327 (+ 68us) server_negotiation.cc:407] Connection header received
0928 09:07:00.370395 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.370396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.370407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.370463 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:00.370649 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.371135 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.372210 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.372589 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.372659 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.372667 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.372853 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.372905 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:07:00.373104 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.369989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34895 (local address 127.0.98.1:45123)
0928 09:07:00.370145 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.370157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.370214 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.370372 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.370373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.370374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:00.370449 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.370451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.371133 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.371134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.371942 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.371944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.372790 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.372791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.372886 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.372890 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.372952 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.372989 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":31}
I20260928 09:07:00.373116 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.370098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45123 (local address 127.0.98.3:34895)
0928 09:07:00.370143 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.370146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.370204 (+ 58us) server_negotiation.cc:407] Connection header received
0928 09:07:00.370275 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.370276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.370293 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.370364 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:00.370523 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.371049 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.372093 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.372685 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.372760 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.372771 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.372985 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.373029 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:07:00.373373 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052f950b61c8493b85915bca16c62cb7" candidate_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408654adb033416fa94b1337568c1049" is_pre_election: true
I20260928 09:07:00.373450 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052f950b61c8493b85915bca16c62cb7" candidate_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5290392e54994aed81218dedb492ae17" is_pre_election: true
I20260928 09:07:00.373569 14415 raft_consensus.cc:2454] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ed33ef255548b2bc91ef32cb5a0686 in term 0.
I20260928 09:07:00.373596 14549 raft_consensus.cc:2454] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ed33ef255548b2bc91ef32cb5a0686 in term 0.
I20260928 09:07:00.373817 14216 leader_election.cc:302] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408654adb033416fa94b1337568c1049, d9ed33ef255548b2bc91ef32cb5a0686; no voters:
I20260928 09:07:00.373993 14620 raft_consensus.cc:2781] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:07:00.374040 14620 raft_consensus.cc:482] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:07:00.374056 14620 raft_consensus.cc:3037] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:00.374722 14620 raft_consensus.cc:504] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.374859 14620 leader_election.cc:288] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 1 election: Requested vote from peers 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895), 408654adb033416fa94b1337568c1049 (127.0.98.2:40525)
I20260928 09:07:00.375033 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052f950b61c8493b85915bca16c62cb7" candidate_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5290392e54994aed81218dedb492ae17"
I20260928 09:07:00.375036 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052f950b61c8493b85915bca16c62cb7" candidate_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408654adb033416fa94b1337568c1049"
I20260928 09:07:00.375131 14415 raft_consensus.cc:3037] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:00.375113 14549 raft_consensus.cc:3037] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:00.375831 14415 raft_consensus.cc:2454] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ed33ef255548b2bc91ef32cb5a0686 in term 1.
I20260928 09:07:00.375998 14549 raft_consensus.cc:2454] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ed33ef255548b2bc91ef32cb5a0686 in term 1.
I20260928 09:07:00.376062 14216 leader_election.cc:302] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408654adb033416fa94b1337568c1049, d9ed33ef255548b2bc91ef32cb5a0686; no voters:
I20260928 09:07:00.376151 14620 raft_consensus.cc:2781] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:07:00.376304 14620 raft_consensus.cc:686] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 LEADER]: Becoming Leader. State: Replica: d9ed33ef255548b2bc91ef32cb5a0686, State: Running, Role: LEADER
I20260928 09:07:00.376410 14620 consensus_queue.cc:237] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.377244 14136 catalog_manager.cc:5495] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } health_report { overall_health: UNKNOWN } } }
I20260928 09:07:00.383272 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.381016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.0.98.1:38043)
0928 09:07:00.381076 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.381078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.381086 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:07:00.381114 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.381115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.381132 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.381189 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:07:00.381343 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.381860 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.382600 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.382790 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.382843 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.382948 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:07:00.383037 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:07:00.383093 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.383165 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.383181 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:07:00.384989 14415 raft_consensus.cc:1261] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Refusing update from remote peer d9ed33ef255548b2bc91ef32cb5a0686: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:00.385051 14549 raft_consensus.cc:1261] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Refusing update from remote peer d9ed33ef255548b2bc91ef32cb5a0686: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:00.385349 14622 consensus_queue.cc:1035] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:00.385479 14620 consensus_queue.cc:1035] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:00.387774 14625 mvcc.cc:205] Tried to move back new op lower bound from 7334241977894264832 to 7334241977862328320. Current Snapshot: MvccSnapshot[applied={T|T < 7334241977894264832}]
I20260928 09:07:00.387984 14626 mvcc.cc:205] Tried to move back new op lower bound from 7334241977894264832 to 7334241977862328320. Current Snapshot: MvccSnapshot[applied={T|T < 7334241977894264832}]
I20260928 09:07:00.388455 14627 mvcc.cc:205] Tried to move back new op lower bound from 7334241977894264832 to 7334241977862328320. Current Snapshot: MvccSnapshot[applied={T|T < 7334241977894264832}]
I20260928 09:07:00.390201 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:00.390311 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:00.391017 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Sending DeleteTablet for 3 replicas of tablet 052f950b61c8493b85915bca16c62cb7
I20260928 09:07:00.391350 14261 tablet_service.cc:1514] Processing DeleteTablet for tablet 052f950b61c8493b85915bca16c62cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57718
I20260928 09:07:00.391366 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet 052f950b61c8493b85915bca16c62cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40378
I20260928 09:07:00.391592 14636 tablet_replica.cc:330] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: stopping tablet replica
I20260928 09:07:00.391733 14636 raft_consensus.cc:2229] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:00.391778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14198
I20260928 09:07:00.391758 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 052f950b61c8493b85915bca16c62cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48648
I20260928 09:07:00.393522 14636 raft_consensus.cc:2258] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:00.393698 14637 tablet_replica.cc:330] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: stopping tablet replica
I20260928 09:07:00.393800 14637 raft_consensus.cc:2229] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:00.393906 14637 raft_consensus.cc:2258] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:00.394069 14636 ts_tablet_manager.cc:1905] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:00.394366 14637 ts_tablet_manager.cc:1905] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:00.396311 14636 ts_tablet_manager.cc:1918] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:00.396389 14636 log.cc:1196] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/052f950b61c8493b85915bca16c62cb7
I20260928 09:07:00.396478 14637 ts_tablet_manager.cc:1918] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:00.396560 14637 log.cc:1196] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/052f950b61c8493b85915bca16c62cb7
I20260928 09:07:00.396667 14636 ts_tablet_manager.cc:1939] T 052f950b61c8493b85915bca16c62cb7 P 5290392e54994aed81218dedb492ae17: Deleting consensus metadata
I20260928 09:07:00.396862 14637 ts_tablet_manager.cc:1939] T 052f950b61c8493b85915bca16c62cb7 P 408654adb033416fa94b1337568c1049: Deleting consensus metadata
I20260928 09:07:00.397106 14121 catalog_manager.cc:4841] TS 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895): tablet 052f950b61c8493b85915bca16c62cb7 (table test-table [id=4e641915409b4794a93b341ff64c5a3d]) successfully deleted
I20260928 09:07:00.397326 14122 catalog_manager.cc:4841] TS 408654adb033416fa94b1337568c1049 (127.0.98.2:40525): tablet 052f950b61c8493b85915bca16c62cb7 (table test-table [id=4e641915409b4794a93b341ff64c5a3d]) successfully deleted
W20260928 09:07:00.397529 14123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38043 (error 108)
I20260928 09:07:00.397802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14332
I20260928 09:07:00.402689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14466
I20260928 09:07:00.403321 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.402805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38043 (local address 127.0.0.1:57736)
0928 09:07:00.403232 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.403275 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
W20260928 09:07:00.404114 14123 catalog_manager.cc:4584] TS d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 052f950b61c8493b85915bca16c62cb7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38043: connect: Connection refused (error 111)
I20260928 09:07:00.407112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14104
I20260928 09:07:00.412492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44509
--webserver_interface=127.0.98.62
--webserver_port=37773
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:00.512862 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:00.513087 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:00.513110 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:00.514369 14639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:07:00.514415 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:00.514428 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:00.514441 14639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:07:00.514454 14639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:07:00.515885 14639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:00.516075 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:00.516251 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:00.524771 14645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.524771 14648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.524960 14639 server_base.cc:1034] running on GCE node
W20260928 09:07:00.524789 14646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.525316 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:00.525693 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:00.526841 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790586420526833 us; error 30 us; skew 500 ppm
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:00.535481 14639 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:07:00.535912 14639 webserver.cc:460] Webserver started at http://127.0.98.62:37773/ using document root <none> and password file <none>
I20260928 09:07:00.536140 14639 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:00.536195 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:00.537518 14639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:07:00.538210 14655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.538383 14639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:07:00.538457 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "4865aae91b704fe5ab0a862347575e1d"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "3a1b2399f4bea1dc94e5ef8c73f6c2fa"
server_key_iv: "d661f16cb5bcbf35c54c02dd8d55ccd4"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.538834 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:00.547082 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:00.548997 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:00.549118 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:00.553675 14639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44509
I20260928 09:07:00.553709 14707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44509 every 8 connection(s)
I20260928 09:07:00.554171 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:07:00.556130 14708 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:07:00.556952 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap starting.
I20260928 09:07:00.557607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14639
I20260928 09:07:00.557966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38043
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44509
--webserver_port=44339
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:00.566350 14708 log.cc:824] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:00.568675 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:00.568988 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap complete.
I20260928 09:07:00.571605 14708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:00.571882 14708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4865aae91b704fe5ab0a862347575e1d, State: Initialized, Role: FOLLOWER
I20260928 09:07:00.572124 14708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:00.572227 14708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:07:00.572484 14708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:07:00.572562 14708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:07:00.573745 14708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:00.573870 14708 leader_election.cc:302] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4865aae91b704fe5ab0a862347575e1d; no voters:
I20260928 09:07:00.574087 14708 leader_election.cc:288] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:07:00.574127 14712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:07:00.574254 14712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 LEADER]: Becoming Leader. State: Replica: 4865aae91b704fe5ab0a862347575e1d, State: Running, Role: LEADER
I20260928 09:07:00.574343 14712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:00.574381 14708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:07:00.574698 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4865aae91b704fe5ab0a862347575e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } } }
I20260928 09:07:00.574782 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:00.574996 14714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4865aae91b704fe5ab0a862347575e1d. Latest consensus state: current_term: 2 leader_uuid: "4865aae91b704fe5ab0a862347575e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } } }
I20260928 09:07:00.575116 14714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:00.575260 14717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:07:00.576026 14717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:07:00.576126 14717 catalog_manager.cc:1259] Loaded cluster ID: 92846394d05e4c26bf752c2de15982cf
I20260928 09:07:00.576396 14717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:07:00.576719 14717 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 09:07:00.582051 14717 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 09:07:00.584824 14717 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:07:00.585045 14717 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Loaded TSK: 0
I20260928 09:07:00.585285 14717 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:07:00.643605 14710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:00.643828 14710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:00.643867 14710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:00.644994 14710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:00.645061 14710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:00.645107 14710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:07:00.646627 14710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44339
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:00.646883 14710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:00.647130 14710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:00.652276 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.652276 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.652433 14710 server_base.cc:1034] running on GCE node
W20260928 09:07:00.652308 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.652848 14710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:00.653167 14710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:00.654348 14710 hybrid_clock.cc:648] HybridClock initialized: now 1790586420654333 us; error 28 us; skew 500 ppm
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:00.657166 14710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:07:00.657763 14710 webserver.cc:460] Webserver started at http://127.0.98.1:44339/ using document root <none> and password file <none>
I20260928 09:07:00.658011 14710 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:00.658106 14710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:00.659508 14710 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:07:00.660205 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.660446 14710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:00.660557 14710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "d9ed33ef255548b2bc91ef32cb5a0686"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "d7ccdfe92d26f2361eef2e7945f2d5a7"
server_key_iv: "679a35c479305649d27bcf903dbf3010"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.661053 14710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:00.688879 14710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:00.691999 14710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:00.692190 14710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:00.692489 14710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:00.693043 14752 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:07:00.694661 14710 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:07:00.694729 14710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:07:00.694770 14710 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:07:00.695768 14710 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:07:00.695823 14710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:00.695900 14752 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap starting.
I20260928 09:07:00.704083 14710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38043
I20260928 09:07:00.704614 14710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:07:00.706413 14752 log.cc:824] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:00.707398 14859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38043 every 8 connection(s)
I20260928 09:07:00.709447 14752 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:00.709869 14752 tablet_bootstrap.cc:492] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap complete.
I20260928 09:07:00.711220 14752 ts_tablet_manager.cc:1397] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:00.713529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14710
I20260928 09:07:00.713881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40525
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44509
--webserver_port=36489
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:00.713783 14752 raft_consensus.cc:348] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.714201 14752 raft_consensus.cc:729] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed33ef255548b2bc91ef32cb5a0686, State: Initialized, Role: FOLLOWER
I20260928 09:07:00.714376 14752 consensus_queue.cc:260] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } } peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } }
I20260928 09:07:00.714761 14752 ts_tablet_manager.cc:1428] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:07:00.718109 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.705339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:45949)
0928 09:07:00.705605 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.705661 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.706317 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.707209 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.707213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.707685 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:00.707876 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.707882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.709059 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.709060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.710201 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.710204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.710839 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.710840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.710976 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.711613 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:00.711622 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:00.713576 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:00.716379 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.716384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.716385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.716619 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.716895 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.716896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.716897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.717006 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.717406 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:00.717412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:00.717597 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.717716 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.717905 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
I20260928 09:07:00.718119 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.705506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45949 (local address 127.0.98.62:44509)
0928 09:07:00.706736 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.706745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.706786 (+ 41us) server_negotiation.cc:407] Connection header received
0928 09:07:00.706928 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.706959 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.707023 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.707175 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.708003 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.708932 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.710301 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.710729 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.710861 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.713756 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.713761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.713762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.713780 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.716270 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.716717 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.716721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.716721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.716823 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.717085 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.717087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.717087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.717331 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.717427 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.717788 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.717916 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":98,"threads_started":1}
I20260928 09:07:00.718968 14860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44509
I20260928 09:07:00.719061 14860 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:00.719331 14860 heartbeater.cc:511] Master 127.0.98.62:44509 requested a full tablet report, sending...
I20260928 09:07:00.720075 14672 ts_manager.cc:194] Registered new tserver with Master: d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043)
I20260928 09:07:00.721316 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45949
I20260928 09:07:00.722344 14860 heartbeater.cc:503] Master 127.0.98.62:44509 was elected leader, sending a full tablet report...
I20260928 09:07:00.725281 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.720707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38043 (local address 127.0.0.1:57740)
0928 09:07:00.721749 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.721783 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.721877 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.722325 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.722328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.722329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:00.722510 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.722513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.723426 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.723426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.724241 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.724243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.724937 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.724938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.725034 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.725046 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.725146 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.725202 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":117,"threads_started":1}
I20260928 09:07:00.725274 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.721122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.0.98.1:38043)
0928 09:07:00.721297 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.721301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.721839 (+ 538us) server_negotiation.cc:407] Connection header received
0928 09:07:00.721977 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.721979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.722190 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.722267 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:00.722610 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.723346 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.724338 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.724812 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.724910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.724972 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.725165 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.725206 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
I20260928 09:07:00.726029 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 052f950b61c8493b85915bca16c62cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57740
I20260928 09:07:00.726524 14869 tablet_replica.cc:330] stopping tablet replica
I20260928 09:07:00.726680 14869 raft_consensus.cc:2229] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:00.726752 14869 raft_consensus.cc:2258] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:00.727062 14869 ts_tablet_manager.cc:1905] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:00.729215 14869 ts_tablet_manager.cc:1918] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:00.729308 14869 log.cc:1196] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/052f950b61c8493b85915bca16c62cb7
I20260928 09:07:00.729530 14869 ts_tablet_manager.cc:1939] T 052f950b61c8493b85915bca16c62cb7 P d9ed33ef255548b2bc91ef32cb5a0686: Deleting consensus metadata
I20260928 09:07:00.730123 14659 catalog_manager.cc:4841] TS d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043): tablet 052f950b61c8493b85915bca16c62cb7 (table test-table [id=4e641915409b4794a93b341ff64c5a3d]) successfully deleted
W20260928 09:07:00.813884 14866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:00.814095 14866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:00.814118 14866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:00.815346 14866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:00.815392 14866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:00.815419 14866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:07:00.816984 14866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36489
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:00.817247 14866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:00.817483 14866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:00.822456 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.822456 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.822536 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.822777 14866 server_base.cc:1034] running on GCE node
I20260928 09:07:00.823006 14866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:00.823331 14866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:00.824520 14866 hybrid_clock.cc:648] HybridClock initialized: now 1790586420824496 us; error 53 us; skew 500 ppm
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:00.827147 14866 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:07:00.827606 14866 webserver.cc:460] Webserver started at http://127.0.98.2:36489/ using document root <none> and password file <none>
I20260928 09:07:00.827855 14866 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:00.827940 14866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:00.829368 14866 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:00.830236 14884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.830498 14866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:07:00.830607 14866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "408654adb033416fa94b1337568c1049"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "b969a49444897ea692190fa126831244"
server_key_iv: "cd8a38bba62ab7191baecc9fc83e5f06"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.830989 14866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:00.849493 14866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:00.851800 14866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:00.851969 14866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:00.852237 14866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:00.852555 14866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:00.852607 14866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.852650 14866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:00.852676 14866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.858700 14866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40525
I20260928 09:07:00.858752 14997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40525 every 8 connection(s)
I20260928 09:07:00.859134 14866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:07:00.862421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 14866
I20260928 09:07:00.862713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34895
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44509
--webserver_port=38995
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:00.869568 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.859888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38235 (local address 127.0.98.62:44509)
0928 09:07:00.859961 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.859964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.860199 (+ 235us) server_negotiation.cc:407] Connection header received
0928 09:07:00.860767 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.860769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.860785 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.860835 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.861867 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.862379 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.863514 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.863804 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.863882 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.866277 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.866279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.866279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.866288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.868063 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.868476 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.868477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.868477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.868543 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.868798 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.868798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.868799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.868957 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.869037 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.869297 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.869402 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:07:00.869537 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.859877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:38235)
0928 09:07:00.860057 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.860099 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.860645 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.860911 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.860914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.861489 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:00.861763 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.861770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.862481 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.862482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.863423 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.863425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.863910 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.863912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.864036 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.864598 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:00.864606 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:00.866180 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:00.868167 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.868174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.868176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.868399 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.868630 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.868631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.868631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.868729 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.869028 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:00.869039 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:00.869163 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.869247 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.869369 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260928 09:07:00.870188 14998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44509
I20260928 09:07:00.870302 14998 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:00.870522 14998 heartbeater.cc:511] Master 127.0.98.62:44509 requested a full tablet report, sending...
I20260928 09:07:00.870950 14672 ts_manager.cc:194] Registered new tserver with Master: 408654adb033416fa94b1337568c1049 (127.0.98.2:40525)
I20260928 09:07:00.871340 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38235
W20260928 09:07:00.948109 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:00.948316 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:00.948339 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:00.949525 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:00.949630 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:00.949659 15001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:07:00.951157 15001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38995
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:00.951421 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:00.951663 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:00.956840 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:00.956840 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.957036 15001 server_base.cc:1034] running on GCE node
W20260928 09:07:00.956874 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:00.957329 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:00.957633 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:00.958813 15001 hybrid_clock.cc:648] HybridClock initialized: now 1790586420958787 us; error 43 us; skew 500 ppm
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:00.961290 15001 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:07:00.961786 15001 webserver.cc:460] Webserver started at http://127.0.98.3:38995/ using document root <none> and password file <none>
I20260928 09:07:00.961978 15001 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:00.962030 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:00.963305 15001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:00.963850 15016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.964002 15001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:07:00.964090 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "5290392e54994aed81218dedb492ae17"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "e20efd37483c3809d6feb7a97d53f398"
server_key_iv: "b6a1c7d83ef579777e47a2bfeb0ae9b3"
server_key_version: "encryptionkey@0"
I20260928 09:07:00.964432 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:00.977133 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:00.979092 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:00.979202 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:00.979378 15001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:00.979624 15001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:00.979652 15001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.979673 15001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:00.979686 15001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:00.985324 15001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34895
I20260928 09:07:00.985400 15129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34895 every 8 connection(s)
I20260928 09:07:00.985704 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:07:00.987923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15001
Sep 28 09:07:00 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:00.995615 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.988339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.0.98.62:44509)
0928 09:07:00.988460 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.988462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.988471 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:07:00.988540 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.988541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.988553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.988656 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.989241 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.989808 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.990780 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.991079 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.991155 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.992517 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.992518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.992519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.992526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.994317 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.994750 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.994751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.994752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.994806 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.995044 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.995045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.995045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.995177 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.995354 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.995438 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.995525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260928 09:07:00.996219 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.986418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34243 (local address 127.0.98.62:44509)
0928 09:07:00.986497 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:00.986500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:00.986733 (+ 233us) server_negotiation.cc:407] Connection header received
0928 09:07:00.987261 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:00.987262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:00.987272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:00.987330 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:00.988370 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.988901 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.989994 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.990277 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.990348 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.992742 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:00.992743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:00.992744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:00.992750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:00.994545 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.995085 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.995086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.995087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.995140 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:00.995431 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:00.995432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:00.995432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:00.995656 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:00.995724 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:00.996057 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 09:07:00.996131 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:07:00.996472 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:00.986406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:34243)
0928 09:07:00.986593 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:00.986635 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:00.987171 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:00.987385 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:00.987388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:00.987970 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:00.988234 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.988239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.989001 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.989002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.989919 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:00.989921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:00.990358 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:00.990359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:00.990458 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:00.991050 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:00.991060 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:00.992640 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:00.994663 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.994671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.994673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.994990 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.995231 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:00.995232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:00.995233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:00.995362 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:00.995723 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:00.995731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:00.995909 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 09:07:00.996025 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 09:07:00.996200 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260928 09:07:00.996971 15130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44509
I20260928 09:07:00.997080 15130 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:00.997282 15130 heartbeater.cc:511] Master 127.0.98.62:44509 requested a full tablet report, sending...
I20260928 09:07:00.997720 14672 ts_manager.cc:194] Registered new tserver with Master: 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895)
I20260928 09:07:00.998103 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34243
I20260928 09:07:01.007460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:07:01.010993 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.007852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.98.62:44509)
0928 09:07:01.007909 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:01.007911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:01.008067 (+ 156us) server_negotiation.cc:407] Connection header received
0928 09:07:01.008200 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:01.008201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:01.008250 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:01.008319 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:07:01.008436 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.009268 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.010158 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.010384 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.010460 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.010653 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:07:01.010767 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:07:01.010865 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:01.010884 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 09:07:01.010905 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:07:01.011384 14672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:07:01.013103 14672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:07:01.015343 14794 tablet_service.cc:1467] Processing CreateTablet for tablet 2111c558be9740c884baa7c46b021b79 (DEFAULT_TABLE table=test-table [id=93b92afc2fb140718003f342c1272fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:01.015552 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2111c558be9740c884baa7c46b021b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:01.017132 14752 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap starting.
I20260928 09:07:01.017464 14752 tablet_bootstrap.cc:654] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:01.018855 14752 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: No bootstrap required, opened a new log
I20260928 09:07:01.018858 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.015090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40525 (local address 127.0.0.1:48652)
0928 09:07:01.015188 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.015200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:01.015244 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:01.015554 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:01.015555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:01.015556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:01.015613 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.015614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.016845 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.016845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.017878 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.017881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.018602 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.018604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.018661 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.018665 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:07:01.018721 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 09:07:01.018784 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":12}
I20260928 09:07:01.018909 14752 ts_tablet_manager.cc:1397] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:07:01.019018 14752 raft_consensus.cc:348] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.019083 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.015228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.0.98.2:40525)
0928 09:07:01.015401 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:01.015406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:01.015416 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:07:01.015457 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:01.015459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:01.015474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:01.015557 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:01.015681 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.016734 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.017969 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.018514 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.018604 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.018675 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:01.018958 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 09:07:01.019003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260928 09:07:01.019096 14752 raft_consensus.cc:374] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:01.019145 14752 raft_consensus.cc:729] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed33ef255548b2bc91ef32cb5a0686, State: Initialized, Role: FOLLOWER
I20260928 09:07:01.019196 14752 consensus_queue.cc:260] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.019312 14752 ts_tablet_manager.cc:1428] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.019876 14932 tablet_service.cc:1467] Processing CreateTablet for tablet 2111c558be9740c884baa7c46b021b79 (DEFAULT_TABLE table=test-table [id=93b92afc2fb140718003f342c1272fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:01.020262 14932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2111c558be9740c884baa7c46b021b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:01.020948 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.015131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34895 (local address 127.0.0.1:40388)
0928 09:07:01.015410 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.015421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:01.015475 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:01.017522 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:01.017523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:01.017526 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:01.017592 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.017594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.018632 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.018633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.019732 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.019735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.020700 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.020702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.020773 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.020777 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:07:01.020834 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:07:01.020875 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260928 09:07:01.020961 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.015374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.0.98.3:34895)
0928 09:07:01.017320 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:01.017325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:01.017335 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:07:01.017391 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:01.017393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:01.017408 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:01.017620 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:01.017692 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.018523 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.019932 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.020599 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.020697 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.020751 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:01.020848 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 09:07:01.020884 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":89,"threads_started":1}
I20260928 09:07:01.021752 15064 tablet_service.cc:1467] Processing CreateTablet for tablet 2111c558be9740c884baa7c46b021b79 (DEFAULT_TABLE table=test-table [id=93b92afc2fb140718003f342c1272fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:01.022038 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2111c558be9740c884baa7c46b021b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:01.023274 15140 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Bootstrap starting.
I20260928 09:07:01.023845 15140 tablet_bootstrap.cc:654] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:01.024325 15140 log.cc:824] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:01.024765 15142 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Bootstrap starting.
I20260928 09:07:01.025199 15140 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: No bootstrap required, opened a new log
I20260928 09:07:01.025269 15140 ts_tablet_manager.cc:1397] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:07:01.025408 15142 tablet_bootstrap.cc:654] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:01.025934 15142 log.cc:824] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:01.026630 15142 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: No bootstrap required, opened a new log
I20260928 09:07:01.026708 15142 ts_tablet_manager.cc:1397] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:07:01.026839 15140 raft_consensus.cc:348] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.026997 15140 raft_consensus.cc:374] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:01.027021 15140 raft_consensus.cc:729] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408654adb033416fa94b1337568c1049, State: Initialized, Role: FOLLOWER
I20260928 09:07:01.027158 15140 consensus_queue.cc:260] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.027441 15140 ts_tablet_manager.cc:1428] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:07:01.027477 14998 heartbeater.cc:503] Master 127.0.98.62:44509 was elected leader, sending a full tablet report...
I20260928 09:07:01.028802 15142 raft_consensus.cc:348] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.028977 15142 raft_consensus.cc:374] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:01.029022 15142 raft_consensus.cc:729] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5290392e54994aed81218dedb492ae17, State: Initialized, Role: FOLLOWER
I20260928 09:07:01.029146 15142 consensus_queue.cc:260] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.029453 15142 ts_tablet_manager.cc:1428] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:07:01.029500 15130 heartbeater.cc:503] Master 127.0.98.62:44509 was elected leader, sending a full tablet report...
W20260928 09:07:01.109753 14999 tablet.cc:2369] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:07:01.148743 15145 raft_consensus.cc:482] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:01.148929 15145 raft_consensus.cc:504] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.149346 15145 leader_election.cc:288] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 408654adb033416fa94b1337568c1049 (127.0.98.2:40525), d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043)
I20260928 09:07:01.152421 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.149632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40725 (local address 127.0.98.2:40525)
0928 09:07:01.149698 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:01.149700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:01.149868 (+ 168us) server_negotiation.cc:407] Connection header received
0928 09:07:01.149899 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:01.149900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:01.149913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:01.149975 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:01.150257 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.150751 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.151705 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.152085 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.152157 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.152167 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:01.152301 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 09:07:01.152347 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 09:07:01.152427 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.149559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38043 (local address 127.0.98.3:53839)
0928 09:07:01.149649 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.149661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:01.149713 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:01.149886 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:01.149887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:01.149889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:01.149969 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.149971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.150719 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.150720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.151564 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.151566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.152156 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.152157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.152244 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.152247 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:07:01.152320 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 09:07:01.152358 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 09:07:01.152452 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.149668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53839 (local address 127.0.98.1:38043)
0928 09:07:01.149710 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:01.149712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:01.149721 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:07:01.149782 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:01.149784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:01.149798 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:01.149861 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:01.150081 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.150637 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.151661 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.152034 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.152087 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.152096 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:01.152343 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 09:07:01.152390 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:07:01.152444 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.149571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40525 (local address 127.0.98.3:40725)
0928 09:07:01.149774 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.149784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:01.149851 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:01.150092 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:01.150093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:01.150095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:01.150172 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.150174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.150842 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.150842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.151594 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:01.151595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.152176 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.152176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:01.152230 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.152233 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:07:01.152276 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 09:07:01.152335 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":51,"thread_start_us":97,"threads_started":1}
I20260928 09:07:01.152724 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111c558be9740c884baa7c46b021b79" candidate_uuid: "5290392e54994aed81218dedb492ae17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" is_pre_election: true
I20260928 09:07:01.152802 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111c558be9740c884baa7c46b021b79" candidate_uuid: "5290392e54994aed81218dedb492ae17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408654adb033416fa94b1337568c1049" is_pre_election: true
I20260928 09:07:01.152922 14952 raft_consensus.cc:2454] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5290392e54994aed81218dedb492ae17 in term 0.
I20260928 09:07:01.152923 14814 raft_consensus.cc:2454] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5290392e54994aed81218dedb492ae17 in term 0.
I20260928 09:07:01.153177 15019 leader_election.cc:302] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408654adb033416fa94b1337568c1049, 5290392e54994aed81218dedb492ae17; no voters:
I20260928 09:07:01.153319 15145 raft_consensus.cc:2781] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:07:01.153384 15145 raft_consensus.cc:482] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:07:01.153411 15145 raft_consensus.cc:3037] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:01.154105 15145 raft_consensus.cc:504] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.154255 15145 leader_election.cc:288] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [CANDIDATE]: Term 1 election: Requested vote from peers 408654adb033416fa94b1337568c1049 (127.0.98.2:40525), d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043)
I20260928 09:07:01.154420 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111c558be9740c884baa7c46b021b79" candidate_uuid: "5290392e54994aed81218dedb492ae17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408654adb033416fa94b1337568c1049"
I20260928 09:07:01.154412 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111c558be9740c884baa7c46b021b79" candidate_uuid: "5290392e54994aed81218dedb492ae17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ed33ef255548b2bc91ef32cb5a0686"
I20260928 09:07:01.154493 14814 raft_consensus.cc:3037] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:01.154517 14952 raft_consensus.cc:3037] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:01.155207 14952 raft_consensus.cc:2454] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5290392e54994aed81218dedb492ae17 in term 1.
I20260928 09:07:01.155421 15019 leader_election.cc:302] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408654adb033416fa94b1337568c1049, 5290392e54994aed81218dedb492ae17; no voters:
I20260928 09:07:01.155581 15145 raft_consensus.cc:2781] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:07:01.155756 15145 raft_consensus.cc:686] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 1 LEADER]: Becoming Leader. State: Replica: 5290392e54994aed81218dedb492ae17, State: Running, Role: LEADER
I20260928 09:07:01.155791 14814 raft_consensus.cc:2454] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5290392e54994aed81218dedb492ae17 in term 1.
I20260928 09:07:01.155870 15145 consensus_queue.cc:237] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.156903 14672 catalog_manager.cc:5495] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5290392e54994aed81218dedb492ae17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5290392e54994aed81218dedb492ae17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } health_report { overall_health: UNKNOWN } } }
I20260928 09:07:01.166926 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.164342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.98.3:34895)
0928 09:07:01.164404 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:01.164405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:01.164414 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:07:01.164460 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:01.164461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:01.164478 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:01.164538 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:07:01.164694 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.165177 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.166030 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:01.166252 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:01.166310 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:01.166503 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:07:01.166595 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:07:01.166696 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:01.166788 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:07:01.166811 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:07:01.168845 14952 raft_consensus.cc:1261] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Refusing update from remote peer 5290392e54994aed81218dedb492ae17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:01.168885 14814 raft_consensus.cc:1261] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Refusing update from remote peer 5290392e54994aed81218dedb492ae17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:01.169237 15145 consensus_queue.cc:1035] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:01.169386 15147 consensus_queue.cc:1035] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:01.171903 15152 mvcc.cc:205] Tried to move back new op lower bound from 7334241981104906240 to 7334241981055254528. Current Snapshot: MvccSnapshot[applied={T|T < 7334241981104906240}]
I20260928 09:07:01.171977 15151 mvcc.cc:205] Tried to move back new op lower bound from 7334241981104906240 to 7334241981055254528. Current Snapshot: MvccSnapshot[applied={T|T < 7334241981104906240}]
I20260928 09:07:01.173939 14671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:01.174045 14671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:01.174726 14671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Sending DeleteTablet for 3 replicas of tablet 2111c558be9740c884baa7c46b021b79
I20260928 09:07:01.175045 15064 tablet_service.cc:1514] Processing DeleteTablet for tablet 2111c558be9740c884baa7c46b021b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40388
I20260928 09:07:01.175185 14932 tablet_service.cc:1514] Processing DeleteTablet for tablet 2111c558be9740c884baa7c46b021b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48652
I20260928 09:07:01.175251 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2111c558be9740c884baa7c46b021b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57740
I20260928 09:07:01.175357 14869 tablet_replica.cc:330] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: stopping tablet replica
I20260928 09:07:01.175366 15160 tablet_replica.cc:330] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: stopping tablet replica
I20260928 09:07:01.175413 14869 raft_consensus.cc:2229] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:01.175463 14869 raft_consensus.cc:2258] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:01.175472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14710
I20260928 09:07:01.175700 15160 raft_consensus.cc:2229] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:07:01.175822 15160 raft_consensus.cc:2258] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:01.176518 15161 tablet_replica.cc:330] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: stopping tablet replica
I20260928 09:07:01.176517 15160 ts_tablet_manager.cc:1905] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:01.176597 15161 raft_consensus.cc:2229] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:01.176672 15161 raft_consensus.cc:2258] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:01.177031 15161 ts_tablet_manager.cc:1905] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:01.178555 15161 ts_tablet_manager.cc:1918] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:01.178632 15161 log.cc:1196] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/2111c558be9740c884baa7c46b021b79
I20260928 09:07:01.178838 15160 ts_tablet_manager.cc:1918] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:01.178931 15161 ts_tablet_manager.cc:1939] T 2111c558be9740c884baa7c46b021b79 P 408654adb033416fa94b1337568c1049: Deleting consensus metadata
I20260928 09:07:01.178959 15160 log.cc:1196] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/2111c558be9740c884baa7c46b021b79
I20260928 09:07:01.179342 15160 ts_tablet_manager.cc:1939] T 2111c558be9740c884baa7c46b021b79 P 5290392e54994aed81218dedb492ae17: Deleting consensus metadata
I20260928 09:07:01.179517 14658 catalog_manager.cc:4841] TS 408654adb033416fa94b1337568c1049 (127.0.98.2:40525): tablet 2111c558be9740c884baa7c46b021b79 (table test-table [id=93b92afc2fb140718003f342c1272fd6]) successfully deleted
I20260928 09:07:01.179831 14657 catalog_manager.cc:4841] TS 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895): tablet 2111c558be9740c884baa7c46b021b79 (table test-table [id=93b92afc2fb140718003f342c1272fd6]) successfully deleted
W20260928 09:07:01.180704 14659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38043 (error 108)
W20260928 09:07:01.181783 15136 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:07:01.181040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38043 (local address 127.0.0.1:57752)
0928 09:07:01.181108 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.181121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:01.181169 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:01.181746 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38043: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 09:07:01.181919 14659 catalog_manager.cc:4584] TS d9ed33ef255548b2bc91ef32cb5a0686 (127.0.98.1:38043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2111c558be9740c884baa7c46b021b79: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38043: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:07:01.182605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14866
I20260928 09:07:01.189962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15001
I20260928 09:07:01.195281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 14639
I20260928 09:07:01.200843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44509
--webserver_interface=127.0.98.62
--webserver_port=37773
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:01.282207 15163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:01.282437 15163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:01.282462 15163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:01.283716 15163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:07:01.283766 15163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:01.283782 15163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:01.283794 15163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:07:01.283807 15163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:07:01.285367 15163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:01.285681 15163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:01.285964 15163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:01.291112 15171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:01.291174 15163 server_base.cc:1034] running on GCE node
W20260928 09:07:01.291249 15168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:01.291086 15169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:01.291540 15163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:01.291873 15163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:01.293074 15163 hybrid_clock.cc:648] HybridClock initialized: now 1790586421293059 us; error 54 us; skew 500 ppm
Sep 28 09:07:01 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:01.295701 15163 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:07:01.296274 15163 webserver.cc:460] Webserver started at http://127.0.98.62:37773/ using document root <none> and password file <none>
I20260928 09:07:01.296476 15163 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:01.296525 15163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:01.297895 15163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:07:01.298578 15178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.298823 15163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:01.298913 15163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "4865aae91b704fe5ab0a862347575e1d"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "3a1b2399f4bea1dc94e5ef8c73f6c2fa"
server_key_iv: "d661f16cb5bcbf35c54c02dd8d55ccd4"
server_key_version: "encryptionkey@0"
I20260928 09:07:01.299268 15163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:01.317111 15163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:01.320143 15163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:01.320335 15163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:01.324983 15163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44509
I20260928 09:07:01.325096 15230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44509 every 8 connection(s)
I20260928 09:07:01.325454 15163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:07:01.325743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15163
I20260928 09:07:01.326071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38043
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44509
--webserver_port=44339
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:01.327061 15231 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:07:01.327634 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap starting.
I20260928 09:07:01.338481 15231 log.cc:824] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:01.341684 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:01.342022 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d: Bootstrap complete.
I20260928 09:07:01.345288 15231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:01.345659 15231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4865aae91b704fe5ab0a862347575e1d, State: Initialized, Role: FOLLOWER
I20260928 09:07:01.345849 15231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:01.345916 15231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:07:01.345963 15231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:07:01.346001 15231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 2 FOLLOWER]: Advancing to term 3
I20260928 09:07:01.347396 15231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:01.347538 15231 leader_election.cc:302] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4865aae91b704fe5ab0a862347575e1d; no voters:
I20260928 09:07:01.347743 15231 leader_election.cc:288] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 09:07:01.347863 15235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260928 09:07:01.348055 15231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:07:01.348081 15235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [term 3 LEADER]: Becoming Leader. State: Replica: 4865aae91b704fe5ab0a862347575e1d, State: Running, Role: LEADER
I20260928 09:07:01.348306 15235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } }
I20260928 09:07:01.348692 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4865aae91b704fe5ab0a862347575e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } } }
I20260928 09:07:01.348784 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:01.348963 15237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4865aae91b704fe5ab0a862347575e1d. Latest consensus state: current_term: 3 leader_uuid: "4865aae91b704fe5ab0a862347575e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4865aae91b704fe5ab0a862347575e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44509 } } }
I20260928 09:07:01.348994 15244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:07:01.349022 15237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4865aae91b704fe5ab0a862347575e1d [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:01.349954 15244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:07:01.350072 15244 catalog_manager.cc:1259] Loaded cluster ID: 92846394d05e4c26bf752c2de15982cf
I20260928 09:07:01.350287 15244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 09:07:01.350545 15244 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fede30689ad google::LogMessage::Fail() at ??:0
@ 0x7fede306b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fede30684de google::LogMessage::Flush() at ??:0
@ 0x7fede3069a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fedeaf890f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fedeaf928b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fede4f360d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fede4f2b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fede5d8f6db start_thread at ??:0
@ 0x7fede1b6661f clone at ??:0
W20260928 09:07:01.451797 15233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:01.452071 15233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:01.452119 15233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:01.453795 15233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:01.453857 15233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:01.453897 15233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:07:01.456041 15233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44339
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:01.456300 15233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:01.456559 15233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:01.461869 15267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:01.461869 15268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:01.461846 15270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:01.462423 15233 server_base.cc:1034] running on GCE node
I20260928 09:07:01.462639 15233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:01.462896 15233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:01.464074 15233 hybrid_clock.cc:648] HybridClock initialized: now 1790586421464060 us; error 31 us; skew 500 ppm
Sep 28 09:07:01 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:01.466930 15233 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:07:01.467487 15233 webserver.cc:460] Webserver started at http://127.0.98.1:44339/ using document root <none> and password file <none>
I20260928 09:07:01.467738 15233 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:01.467810 15233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:01.469250 15233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:01.470155 15277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.470378 15233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:01.470487 15233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "d9ed33ef255548b2bc91ef32cb5a0686"
format_stamp: "Formatted at 2026-09-28 09:06:59 on dist-test-slave-smns"
server_key: "d7ccdfe92d26f2361eef2e7945f2d5a7"
server_key_iv: "679a35c479305649d27bcf903dbf3010"
server_key_version: "encryptionkey@0"
I20260928 09:07:01.470878 15233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:01.481962 15233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:01.484387 15233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:01.484557 15233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:01.484812 15233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:01.485340 15284 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:07:01.487463 15233 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:07:01.487536 15233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:07:01.487582 15233 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:07:01.488250 15233 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:07:01.488309 15233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:01.488384 15284 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap starting.
I20260928 09:07:01.495855 15233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38043
I20260928 09:07:01.495937 15391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38043 every 8 connection(s)
I20260928 09:07:01.496336 15233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:07:01.497586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15233
I20260928 09:07:01.497855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40525
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44509
--webserver_port=36489
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:01.499980 15284 log.cc:824] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:01.503003 15284 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:01.503407 15284 tablet_bootstrap.cc:492] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Bootstrap complete.
I20260928 09:07:01.504761 15284 ts_tablet_manager.cc:1397] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260928 09:07:01.506685 15284 raft_consensus.cc:348] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.507130 15284 raft_consensus.cc:729] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed33ef255548b2bc91ef32cb5a0686, State: Initialized, Role: FOLLOWER
I20260928 09:07:01.507427 15284 consensus_queue.cc:260] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:01.507802 15284 ts_tablet_manager.cc:1428] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260928 09:07:01.585798 15394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:07:01.497069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:33633)
0928 09:07:01.497243 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.497287 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:01.498416 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:01.585595 (+ 87179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
W20260928 09:07:01.586084 15392 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44509 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44509: Client connection negotiation failed: client connection to 127.0.98.62:44509: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:07:01.586449 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.586328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:48287)
0928 09:07:01.586381 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.586420 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 09:07:01.586304 15395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:01.586511 15395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:01.586581 15395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 09:07:01.586829 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.586728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:51933)
0928 09:07:01.586776 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.586802 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 09:07:01.586935 15392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:07:01.587981 15395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:01.588054 15395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:01.588100 15395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:07:01.589685 15395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36489
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:01.589960 15395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:01.590205 15395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:07:01.595496 15395 server_base.cc:1034] running on GCE node
W20260928 09:07:01.595420 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:01.595414 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:01.595422 15405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:01.595950 15395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:01.596258 15395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:01.597435 15395 hybrid_clock.cc:648] HybridClock initialized: now 1790586421597426 us; error 41 us; skew 500 ppm
Sep 28 09:07:01 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:01.599994 15395 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:07:01.600463 15395 webserver.cc:460] Webserver started at http://127.0.98.2:36489/ using document root <none> and password file <none>
I20260928 09:07:01.600709 15395 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:01.600782 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:01.602300 15395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:07:01.602916 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.603101 15395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:07:01.603211 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "408654adb033416fa94b1337568c1049"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "b969a49444897ea692190fa126831244"
server_key_iv: "cd8a38bba62ab7191baecc9fc83e5f06"
server_key_version: "encryptionkey@0"
I20260928 09:07:01.603616 15395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:01.618978 15395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:01.621078 15395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:01.621250 15395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:01.621500 15395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:01.621825 15395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:01.621881 15395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.621923 15395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:01.621971 15395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.628278 15395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40525
I20260928 09:07:01.628329 15525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40525 every 8 connection(s)
I20260928 09:07:01.628677 15395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:07:01.629853 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.629402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:60417)
0928 09:07:01.629613 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.629707 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
W20260928 09:07:01.630295 15526 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44509 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44509: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
I20260928 09:07:01.630663 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.630489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:55169)
0928 09:07:01.630603 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.630641 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 09:07:01.631003 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.630897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:44909)
0928 09:07:01.630952 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.630984 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 09:07:01.631177 15526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:07:01.633946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15395
I20260928 09:07:01.634244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34895
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44509
--webserver_port=38995
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:01.719386 15529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:01.719605 15529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:01.719630 15529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:01.720933 15529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:01.720986 15529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:01.721014 15529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:07:01.722551 15529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38995
--tserver_master_addrs=127.0.98.62:44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:01.722821 15529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:01.723071 15529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:01.728451 15535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:01.728466 15529 server_base.cc:1034] running on GCE node
W20260928 09:07:01.728451 15537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:01.728449 15534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:01.729046 15529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:01.729310 15529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:01.730489 15529 hybrid_clock.cc:648] HybridClock initialized: now 1790586421730483 us; error 38 us; skew 500 ppm
Sep 28 09:07:01 dist-test-slave-smns krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:01.733100 15529 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:07:01.733677 15529 webserver.cc:460] Webserver started at http://127.0.98.3:38995/ using document root <none> and password file <none>
I20260928 09:07:01.733928 15529 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:01.733999 15529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:01.735430 15529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:07:01.736078 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.736255 15529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:01.736356 15529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "5290392e54994aed81218dedb492ae17"
format_stamp: "Formatted at 2026-09-28 09:07:00 on dist-test-slave-smns"
server_key: "e20efd37483c3809d6feb7a97d53f398"
server_key_iv: "b6a1c7d83ef579777e47a2bfeb0ae9b3"
server_key_version: "encryptionkey@0"
I20260928 09:07:01.736738 15529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:01.746312 15529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:01.748792 15529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:01.748991 15529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:01.749194 15529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:01.749452 15529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:01.749481 15529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.749534 15529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:01.749615 15529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:01.755316 15529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34895
I20260928 09:07:01.755374 15657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34895 every 8 connection(s)
I20260928 09:07:01.755769 15529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:07:01.756937 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.756550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:50323)
0928 09:07:01.756706 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.756793 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20260928 09:07:01.757385 15658 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44509 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44509: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
I20260928 09:07:01.757714 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.757617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:52769)
0928 09:07:01.757666 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.757694 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 09:07:01.757984 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:01.757911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:53667)
0928 09:07:01.757950 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:01.757970 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:07:01.758102 15658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:07:01.759168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15529
I20260928 09:07:02.587590 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:02.587360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:55571)
0928 09:07:02.587488 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:02.587526 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260928 09:07:02.631762 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:02.631538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:60405)
0928 09:07:02.631662 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:02.631701 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260928 09:07:02.758687 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:02.758444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:36961)
0928 09:07:02.758567 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:02.758609 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260928 09:07:03.016887 15665 raft_consensus.cc:482] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:03.017025 15665 raft_consensus.cc:504] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:03.017513 15665 leader_election.cc:288] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 408654adb033416fa94b1337568c1049 (127.0.98.2:40525), 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895)
Sep 28 09:07:03 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 09:07:03 dist-test-slave-smns krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 09:07:03.029060 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.017877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33757 (local address 127.0.98.3:34895)
0928 09:07:03.018093 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:03.018107 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:03.018145 (+ 38us) server_negotiation.cc:407] Connection header received
0928 09:07:03.018749 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:03.018753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:03.018820 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:03.018990 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:03.020056 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.021011 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.022158 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.022568 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.022704 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:03.025194 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:03.025199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:03.025199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:03.025229 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:03.027280 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:03.027801 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:03.027811 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:03.027812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:03.027907 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:03.028182 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:03.028184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:03.028184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:03.028412 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:03.028506 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:03.028799 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 09:07:03.028931 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260928 09:07:03.029048 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.017727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34895 (local address 127.0.98.1:33757)
0928 09:07:03.017788 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:03.017802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:03.017869 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:03.019176 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:03.019180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:03.019640 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:03.019860 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:03.019867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.021174 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.021175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:03.022060 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:03.022063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.022667 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.022668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:03.022801 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:03.023452 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:03.023456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:03.025062 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:03.027393 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:03.027401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:03.027403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:03.027697 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:03.027986 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:03.027987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:03.027987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:03.028106 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:03.028485 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:03.028500 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:03.028634 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 09:07:03.028744 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 09:07:03.028943 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":22}
I20260928 09:07:03.029175 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.017773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40525 (local address 127.0.98.1:53925)
0928 09:07:03.017941 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:03.017953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:03.018008 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:03.019176 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:03.019179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:03.019639 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:03.019860 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:03.019867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.021136 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.021138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:03.022061 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:03.022063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.022670 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.022671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:03.022793 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:03.023392 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:03.023401 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:03.025417 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:03.027416 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:03.027417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:03.027418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:03.027700 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:03.027979 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:03.027979 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:03.027980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:03.028103 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:03.028486 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:03.028492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:03.028633 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 09:07:03.028744 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 09:07:03.029027 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":198,"thread_start_us":64,"threads_started":1}
I20260928 09:07:03.029083 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.017967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53925 (local address 127.0.98.2:40525)
0928 09:07:03.018109 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:03.018116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:03.018144 (+ 28us) server_negotiation.cc:407] Connection header received
0928 09:07:03.018749 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:03.018753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:03.018820 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:03.018990 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:03.020009 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.021034 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.022156 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:03.022568 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:03.022704 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:03.025537 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:03.025582 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:03.025583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:03.025600 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:03.027323 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:03.027802 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:03.027805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:03.027805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:03.027906 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:03.028172 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:03.028173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:03.028174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:03.028412 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:03.028506 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:03.028799 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 09:07:03.028928 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":50,"threads_started":1}
W20260928 09:07:03.030054 15279 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
W20260928 09:07:03.030125 15280 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 408654adb033416fa94b1337568c1049 (127.0.98.2:40525): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
I20260928 09:07:03.030162 15279 leader_election.cc:302] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9ed33ef255548b2bc91ef32cb5a0686; no voters: 408654adb033416fa94b1337568c1049, 5290392e54994aed81218dedb492ae17
I20260928 09:07:03.030277 15665 raft_consensus.cc:2726] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:07:03.588379 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.588130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:58245)
0928 09:07:03.588276 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:03.588317 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260928 09:07:03.632535 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.632314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:59599)
0928 09:07:03.632446 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:03.632489 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260928 09:07:03.759557 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:03.759315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:46343)
0928 09:07:03.759467 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:03.759512 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 09:07:04.589111 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:04.588882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:36323)
0928 09:07:04.589021 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:04.589060 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260928 09:07:04.633276 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:04.633039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:55027)
0928 09:07:04.633191 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:04.633233 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260928 09:07:04.760381 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:04.760174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:36491)
0928 09:07:04.760299 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:04.760340 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260928 09:07:05.116881 15675 raft_consensus.cc:482] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:05.117013 15675 raft_consensus.cc:504] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:05.117244 15675 leader_election.cc:288] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 408654adb033416fa94b1337568c1049 (127.0.98.2:40525), 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895)
W20260928 09:07:05.117762 15279 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
W20260928 09:07:05.117836 15280 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 408654adb033416fa94b1337568c1049 (127.0.98.2:40525): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
I20260928 09:07:05.117874 15280 leader_election.cc:302] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9ed33ef255548b2bc91ef32cb5a0686; no voters: 408654adb033416fa94b1337568c1049, 5290392e54994aed81218dedb492ae17
I20260928 09:07:05.117961 15675 raft_consensus.cc:2726] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:07:05.589814 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:05.589588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:43927)
0928 09:07:05.589727 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:05.589766 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260928 09:07:05.634080 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:05.633849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:41819)
0928 09:07:05.633984 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:05.634032 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260928 09:07:05.761080 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:05.760853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:60543)
0928 09:07:05.761000 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:05.761039 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260928 09:07:06.590534 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:06.590329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:46679)
0928 09:07:06.590451 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:06.590487 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 09:07:06.634814 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:06.634581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:55081)
0928 09:07:06.634728 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:06.634768 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 09:07:06.761817 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:06.761536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:59629)
0928 09:07:06.761731 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:06.761774 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260928 09:07:07.591375 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:07.591122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:46295)
0928 09:07:07.591280 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:07.591325 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260928 09:07:07.635648 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:07.635390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:46441)
0928 09:07:07.635562 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:07.635601 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260928 09:07:07.674109 15684 raft_consensus.cc:482] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:07.674265 15684 raft_consensus.cc:504] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:07.674464 15684 leader_election.cc:288] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 408654adb033416fa94b1337568c1049 (127.0.98.2:40525), 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895)
W20260928 09:07:07.674924 15279 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
W20260928 09:07:07.674991 15280 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 408654adb033416fa94b1337568c1049 (127.0.98.2:40525): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
I20260928 09:07:07.675024 15280 leader_election.cc:302] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9ed33ef255548b2bc91ef32cb5a0686; no voters: 408654adb033416fa94b1337568c1049, 5290392e54994aed81218dedb492ae17
I20260928 09:07:07.675092 15684 raft_consensus.cc:2726] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:07:07.762482 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:07.762275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:48339)
0928 09:07:07.762404 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:07.762439 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 09:07:08.592173 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:08.591897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:50389)
0928 09:07:08.592078 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:08.592122 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260928 09:07:08.636379 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:08.636181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:41289)
0928 09:07:08.636300 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:08.636334 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 09:07:08.763330 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:08.763099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:50381)
0928 09:07:08.763250 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:08.763286 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 09:07:09.593004 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:09.592762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:46547)
0928 09:07:09.592909 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:09.592955 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260928 09:07:09.637288 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:09.636904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:60559)
0928 09:07:09.637144 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:09.637221 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":66,"threads_started":1}
I20260928 09:07:09.764132 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:09.763892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:39661)
0928 09:07:09.764038 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:09.764083 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 09:07:10.593837 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:10.593622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:59293)
0928 09:07:10.593752 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:10.593789 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 09:07:10.638025 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:10.637814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:48885)
0928 09:07:10.637939 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:10.637980 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260928 09:07:10.764955 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:10.764734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:33081)
0928 09:07:10.764864 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:10.764907 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 09:07:11.594606 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:11.594391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:49953)
0928 09:07:11.594520 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:11.594557 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260928 09:07:11.638772 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:11.638505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:48733)
0928 09:07:11.638666 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:11.638704 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260928 09:07:11.765719 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:11.765454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:56067)
0928 09:07:11.765632 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:11.765671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
I20260928 09:07:12.577220 15698 raft_consensus.cc:482] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:12.577356 15698 raft_consensus.cc:504] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408654adb033416fa94b1337568c1049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40525 } } peers { permanent_uuid: "5290392e54994aed81218dedb492ae17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34895 } } peers { permanent_uuid: "d9ed33ef255548b2bc91ef32cb5a0686" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38043 } }
I20260928 09:07:12.577621 15698 leader_election.cc:288] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 408654adb033416fa94b1337568c1049 (127.0.98.2:40525), 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895)
W20260928 09:07:12.578020 15280 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 408654adb033416fa94b1337568c1049 (127.0.98.2:40525): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
W20260928 09:07:12.578106 15279 leader_election.cc:341] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5290392e54994aed81218dedb492ae17 (127.0.98.3:34895): Not found: Tablet not found: 2111c558be9740c884baa7c46b021b79
I20260928 09:07:12.578172 15279 leader_election.cc:302] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9ed33ef255548b2bc91ef32cb5a0686; no voters: 408654adb033416fa94b1337568c1049, 5290392e54994aed81218dedb492ae17
I20260928 09:07:12.578258 15698 raft_consensus.cc:2726] T 2111c558be9740c884baa7c46b021b79 P d9ed33ef255548b2bc91ef32cb5a0686 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:07:12.595408 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:12.595192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:52229)
0928 09:07:12.595319 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:12.595360 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 09:07:12.639546 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:12.639222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:40093)
0928 09:07:12.639436 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:12.639495 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
I20260928 09:07:12.766481 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:12.766254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:51237)
0928 09:07:12.766390 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:12.766435 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 09:07:13.596199 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:13.595897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:47577)
0928 09:07:13.596111 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:13.596151 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260928 09:07:13.640347 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:13.640085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:57629)
0928 09:07:13.640241 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:13.640298 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260928 09:07:13.767246 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:13.767009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:59841)
0928 09:07:13.767162 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:13.767200 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 09:07:14.596879 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:14.596669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:33427)
0928 09:07:14.596800 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:14.596834 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260928 09:07:14.641096 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:14.640840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:59491)
0928 09:07:14.640963 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:14.641000 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260928 09:07:14.767987 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:14.767746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:33697)
0928 09:07:14.767902 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:14.767939 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260928 09:07:15.597699 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:15.597416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:37075)
0928 09:07:15.597586 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:15.597628 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260928 09:07:15.641937 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:15.641668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:57841)
0928 09:07:15.641809 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:15.641874 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 09:07:15.768716 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:15.768491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:43045)
0928 09:07:15.768631 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:15.768670 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260928 09:07:16.598616 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:16.598384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.1:47207)
0928 09:07:16.598522 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:16.598563 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 09:07:16.642717 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:16.642488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.2:43873)
0928 09:07:16.642628 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:16.642672 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260928 09:07:16.769518 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:16.769294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44509 (local address 127.0.98.3:37143)
0928 09:07:16.769432 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:16.769472 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 09:07:16.769942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15233
I20260928 09:07:16.774818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15395
I20260928 09:07:16.779220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15529
2026-09-28T09:07:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18289 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:07:16 dist-test-slave-smns krb5kdc[15716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:07:16 dist-test-slave-smns krb5kdc[15716](info): set up 2 sockets
Sep 28 09:07:16 dist-test-slave-smns krb5kdc[15716](info): commencing operation
krb5kdc: starting...
W20260928 09:07:17.841696 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:07:17 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:07:17.964859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:18.050274 15729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:18.050536 15729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:18.050589 15729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:18.051949 15729 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 09:07:18.052006 15729 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 09:07:18.052026 15729 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 09:07:18.052044 15729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:07:18.052073 15729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:18.052103 15729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:18.052132 15729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:07:18.052158 15729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:07:18.053740 15729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:18.054016 15729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:18.054258 15729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:18.059986 15734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:18.059979 15737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.059958 15729 server_base.cc:1034] running on GCE node
W20260928 09:07:18.059979 15735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.060420 15729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:18.060678 15729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:18.061852 15729 hybrid_clock.cc:648] HybridClock initialized: now 1790586438061837 us; error 28 us; skew 500 ppm
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:18.064353 15729 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:07:18.064858 15729 webserver.cc:460] Webserver started at http://127.0.98.62:35321/ using document root <none> and password file <none>
I20260928 09:07:18.065148 15729 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:18.065222 15729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:18.065369 15729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:07:18.066501 15729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "201b005f2f774cb491b24adc23d9db24"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "583fdac320ad3806afc626096072f0f6"
server_key_iv: "9b0448ff930955fa30de278d24fb1d89"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.066874 15729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "201b005f2f774cb491b24adc23d9db24"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "583fdac320ad3806afc626096072f0f6"
server_key_iv: "9b0448ff930955fa30de278d24fb1d89"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.068274 15729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:07:18.069123 15744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.069342 15729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:18.069447 15729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "201b005f2f774cb491b24adc23d9db24"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "583fdac320ad3806afc626096072f0f6"
server_key_iv: "9b0448ff930955fa30de278d24fb1d89"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.069535 15729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:18.085909 15729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:18.088100 15729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:18.088318 15729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:18.093041 15729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43785
I20260928 09:07:18.093087 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43785 every 8 connection(s)
I20260928 09:07:18.093503 15729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:07:18.094461 15797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:18.096940 15797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap starting.
I20260928 09:07:18.097609 15797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:18.097906 15797 log.cc:824] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:18.098631 15797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: No bootstrap required, opened a new log
I20260928 09:07:18.100077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15729
I20260928 09:07:18.100199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:07:18.100407 392 external_mini_cluster.cc:1428] Setting key 7215f0e90a87122c85ec0c234a58dadc
I20260928 09:07:18.100543 15797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:18.100740 15797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:18.100827 15797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201b005f2f774cb491b24adc23d9db24, State: Initialized, Role: FOLLOWER
I20260928 09:07:18.101076 15797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:18.101186 15797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:07:18.101225 15797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:07:18.101284 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:18.102617 15797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:18.102753 15797 leader_election.cc:302] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201b005f2f774cb491b24adc23d9db24; no voters:
I20260928 09:07:18.102967 15797 leader_election.cc:288] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:07:18.103004 15802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:07:18.103319 15802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 LEADER]: Becoming Leader. State: Replica: 201b005f2f774cb491b24adc23d9db24, State: Running, Role: LEADER
I20260928 09:07:18.103449 15797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:07:18.103498 15802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:18.103993 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "201b005f2f774cb491b24adc23d9db24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260928 09:07:18.104095 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:18.104545 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201b005f2f774cb491b24adc23d9db24. Latest consensus state: current_term: 1 leader_uuid: "201b005f2f774cb491b24adc23d9db24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260928 09:07:18.104650 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:18.104918 15810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:07:18.105670 15810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:07:18.107528 15810 catalog_manager.cc:1347] Generated new cluster ID: 706b92f67992436092efac4d3fbb2aa5
I20260928 09:07:18.107579 15810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:18.113940 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.100912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.0.98.62:43785)
0928 09:07:18.101105 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.101115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.101152 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:07:18.101277 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.101285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.101343 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.101477 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:18.102455 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.103865 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.105123 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.105743 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.105886 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.109394 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:18.109398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:18.109399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:18.109411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:18.111467 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.111914 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.111917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.111918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.112387 (+ 469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.112645 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.112646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.112646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.112956 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:18.113157 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.113527 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.113731 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260928 09:07:18.140074 15810 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:07:18.140846 15810 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:07:18.149849 15810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Generated new TSK 0
I20260928 09:07:18.150076 15810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:07:18.177306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43785
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:18.260155 15823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:18.260360 15823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:18.260385 15823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:18.261653 15823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:18.261694 15823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:18.261722 15823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:07:18.263242 15823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:18.263491 15823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:18.263762 15823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:18.268899 15828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:18.268932 15829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.269048 15823 server_base.cc:1034] running on GCE node
W20260928 09:07:18.268911 15831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.269490 15823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:18.269871 15823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:18.271049 15823 hybrid_clock.cc:648] HybridClock initialized: now 1790586438271026 us; error 39 us; skew 500 ppm
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:18.273877 15823 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:07:18.274431 15823 webserver.cc:460] Webserver started at http://127.0.98.1:45607/ using document root <none> and password file <none>
I20260928 09:07:18.274670 15823 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:18.274742 15823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:18.274888 15823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:07:18.276011 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b5e5d0eb258464896d8da3789dc3adb"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7c607bfd47158eb699facedb29ac1482"
server_key_iv: "7e02546d09ee080c15ba53e1bf6a512a"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.276408 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b5e5d0eb258464896d8da3789dc3adb"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7c607bfd47158eb699facedb29ac1482"
server_key_iv: "7e02546d09ee080c15ba53e1bf6a512a"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.277904 15823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:07:18.278714 15838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.278937 15823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:18.279049 15823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "4b5e5d0eb258464896d8da3789dc3adb"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7c607bfd47158eb699facedb29ac1482"
server_key_iv: "7e02546d09ee080c15ba53e1bf6a512a"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.279129 15823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:18.304436 15823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:18.306895 15823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:18.307092 15823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:18.307351 15823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:18.307688 15823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:18.307744 15823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.307780 15823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:18.307827 15823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.314841 15823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37377
I20260928 09:07:18.314909 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37377 every 8 connection(s)
I20260928 09:07:18.315413 15823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:07:18.322538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15823
I20260928 09:07:18.322671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:07:18.322873 392 external_mini_cluster.cc:1428] Setting key 564a51d76d3fa49cb3d0e4f103863ea8
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:18.327020 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.316209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39821 (local address 127.0.98.62:43785)
0928 09:07:18.316278 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.316280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.316489 (+ 209us) server_negotiation.cc:407] Connection header received
0928 09:07:18.317304 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.317313 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.317337 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.317450 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:18.318752 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.319260 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.320896 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.321119 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.321190 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.323822 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:18.323823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:18.323823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:18.323832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:18.325423 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.325946 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.325947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.325947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.326009 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.326313 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.326314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.326315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.326428 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:18.326541 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.326828 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.326915 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:07:18.327246 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.316193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:39821)
0928 09:07:18.316359 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.316404 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.317165 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.317571 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.317576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.318318 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:18.318622 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.318630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.319387 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.319389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.320800 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.320803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.321204 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.321205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.321315 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.321893 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:18.321902 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:18.323707 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:18.325517 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:18.325532 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:18.325534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:18.325875 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:18.326105 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:18.326107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:18.326107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:18.326231 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:18.326537 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:18.326544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:18.326688 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.326794 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.326959 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 09:07:18.327730 15952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43785
I20260928 09:07:18.327849 15952 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:18.328032 15952 heartbeater.cc:511] Master 127.0.98.62:43785 requested a full tablet report, sending...
I20260928 09:07:18.328574 15761 ts_manager.cc:194] Registered new tserver with Master: 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377)
I20260928 09:07:18.329185 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39821
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:07:18.350482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43785
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:18.434806 15957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:18.435034 15957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:18.435060 15957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:18.436255 15957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:18.436301 15957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:18.436328 15957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:07:18.437812 15957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:18.438067 15957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:18.438308 15957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:18.443562 15962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:18.443600 15963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.443605 15957 server_base.cc:1034] running on GCE node
W20260928 09:07:18.443575 15965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.443967 15957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:18.444234 15957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:18.445391 15957 hybrid_clock.cc:648] HybridClock initialized: now 1790586438445370 us; error 44 us; skew 500 ppm
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:18.448026 15957 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:07:18.448415 15957 webserver.cc:460] Webserver started at http://127.0.98.2:36477/ using document root <none> and password file <none>
I20260928 09:07:18.448603 15957 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:18.448644 15957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:18.448753 15957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:07:18.449849 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7a3490f4dc0147f135ecbaa3767085ce"
server_key_iv: "18467daf9fd6e4fc34be8c49a6ad8650"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.450157 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7a3490f4dc0147f135ecbaa3767085ce"
server_key_iv: "18467daf9fd6e4fc34be8c49a6ad8650"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.451422 15957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:18.452227 15972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.452426 15957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:18.452507 15957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7a3490f4dc0147f135ecbaa3767085ce"
server_key_iv: "18467daf9fd6e4fc34be8c49a6ad8650"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.452560 15957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:18.465027 15957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:18.467156 15957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:18.467268 15957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:18.467444 15957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:18.467783 15957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:18.467818 15957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.467841 15957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:18.467888 15957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.473994 15957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39183
I20260928 09:07:18.474069 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39183 every 8 connection(s)
I20260928 09:07:18.474409 15957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:07:18.475504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 15957
I20260928 09:07:18.475602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:07:18.475827 392 external_mini_cluster.cc:1428] Setting key 501ebadef62b6ddb1fc690895c5aafe4
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:07:18.485391 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.475170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60907 (local address 127.0.98.62:43785)
0928 09:07:18.475221 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.475224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.475436 (+ 212us) server_negotiation.cc:407] Connection header received
0928 09:07:18.476181 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.476183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.476195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.476251 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:18.477233 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.477956 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.479017 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.479371 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.479445 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.482056 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:18.482058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:18.482058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:18.482067 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:18.483937 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.484399 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.484401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.484401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.484465 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.484780 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.484781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.484781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.484900 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:18.484960 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.485251 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.485323 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:07:18.485634 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.475136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:60907)
0928 09:07:18.475288 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.475346 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.475998 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.476335 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.476339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.476922 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:18.477126 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.477130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.478058 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.478060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.478934 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.478937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.479459 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.479460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.479587 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.480216 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:18.480225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:18.481949 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:18.484075 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:18.484081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:18.484083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:18.484314 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:18.484561 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:18.484562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:18.484563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:18.484706 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:18.484992 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:18.484998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:18.485129 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.485220 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.485346 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260928 09:07:18.486256 16086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43785
I20260928 09:07:18.486344 16086 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:18.486495 16086 heartbeater.cc:511] Master 127.0.98.62:43785 requested a full tablet report, sending...
I20260928 09:07:18.486881 15761 ts_manager.cc:194] Registered new tserver with Master: 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183)
I20260928 09:07:18.487355 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:07:18.504242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43785
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:18.586920 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:18.587142 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:18.587167 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:18.588245 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:18.588280 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:18.588306 16091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:07:18.589757 16091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:18.589931 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:18.590102 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:18.594930 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:18.594930 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:18.594933 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:18.595295 16091 server_base.cc:1034] running on GCE node
I20260928 09:07:18.595538 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:18.595785 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:18.596973 16091 hybrid_clock.cc:648] HybridClock initialized: now 1790586438596941 us; error 42 us; skew 500 ppm
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:18.599745 16091 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:07:18.600303 16091 webserver.cc:460] Webserver started at http://127.0.98.3:41411/ using document root <none> and password file <none>
I20260928 09:07:18.600544 16091 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:18.600615 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:18.600767 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:07:18.601924 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "30baa5450fe84daeb490c0e31f2f7eb8"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "9755dc11748db9c260b2034ea542b6ab"
server_key_iv: "fe86b677756f1955e0f9f48e16879ef9"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.602288 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30baa5450fe84daeb490c0e31f2f7eb8"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "9755dc11748db9c260b2034ea542b6ab"
server_key_iv: "fe86b677756f1955e0f9f48e16879ef9"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.603762 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:07:18.604737 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.604919 16091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:18.605023 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "30baa5450fe84daeb490c0e31f2f7eb8"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "9755dc11748db9c260b2034ea542b6ab"
server_key_iv: "fe86b677756f1955e0f9f48e16879ef9"
server_key_version: "encryptionkey@0"
I20260928 09:07:18.605171 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:18.615389 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:18.617389 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:18.617535 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:18.617834 16091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:18.618129 16091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:18.618162 16091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.618203 16091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:18.618230 16091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:18.623857 16091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36065
I20260928 09:07:18.623934 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36065 every 8 connection(s)
I20260928 09:07:18.624239 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:07:18.629096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16091
I20260928 09:07:18.629189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:07:18.629369 392 external_mini_cluster.cc:1428] Setting key bd7ff63b5ea793e84a9829648f689c81
Sep 28 09:07:18 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:18.633421 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.624941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36467 (local address 127.0.98.62:43785)
0928 09:07:18.625005 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.625009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.625268 (+ 259us) server_negotiation.cc:407] Connection header received
0928 09:07:18.625822 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.625823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.625833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.625890 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:18.626823 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.627346 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.628322 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.628504 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.628557 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.630846 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:18.630848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:18.630848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:18.630855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:18.632118 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.632496 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.632496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.632497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.632550 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.632832 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.632833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.632833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.632948 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:18.633005 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.633291 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.633357 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:07:18.633625 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.624922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:36467)
0928 09:07:18.625142 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.625184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.625732 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.625941 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.625944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.626534 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:18.626728 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.626732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.627429 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.627430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.628244 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.628246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.628577 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.628578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.628677 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.629131 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:18.629141 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:18.630735 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:18.632203 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:18.632209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:18.632211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:18.632430 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:18.632630 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:18.632630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:18.632630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:18.632750 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:18.633014 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:18.633019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:18.633170 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.633261 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.633390 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260928 09:07:18.634081 16220 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43785
I20260928 09:07:18.634184 16220 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:18.634363 16220 heartbeater.cc:511] Master 127.0.98.62:43785 requested a full tablet report, sending...
I20260928 09:07:18.634764 15761 ts_manager.cc:194] Registered new tserver with Master: 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
I20260928 09:07:18.635170 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36467
I20260928 09:07:18.641284 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:07:18.648989 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.642777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.0.98.62:43785)
0928 09:07:18.642840 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.642842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.642973 (+ 131us) server_negotiation.cc:407] Connection header received
0928 09:07:18.643027 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.643028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.643039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.643088 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:18.643702 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.644206 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.644902 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.645101 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.645158 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.646458 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:18.646459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:18.646460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:18.646466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:18.647787 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.648109 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.648109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.648110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.648156 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:18.648468 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:18.648469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:18.648469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:18.648624 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:18.648744 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.648825 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.648884 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:07:18.651087 15761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:07:18.652616 15761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:07:18.659319 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.655126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39183 (local address 127.0.0.1:44568)
0928 09:07:18.655337 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.655378 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.655442 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.655826 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.655829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.655832 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:18.655913 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.655916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.656943 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.656944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.658107 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.658109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.659011 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.659012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.659102 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.659115 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.659182 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.659233 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260928 09:07:18.659580 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.655370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.0.98.3:36065)
0928 09:07:18.655530 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.655535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.655641 (+ 106us) server_negotiation.cc:407] Connection header received
0928 09:07:18.655763 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.655765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.655782 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.655855 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:18.656058 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.657252 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.658465 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.659086 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.659151 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.659202 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.659296 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.659337 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260928 09:07:18.659646 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.655224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.0.98.2:39183)
0928 09:07:18.655568 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.655572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.655583 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:07:18.655627 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.655629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.655646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.655732 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:18.656001 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.656840 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.658199 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.658928 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.659011 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.659082 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.659493 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.659542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":80,"threads_started":1}
I20260928 09:07:18.659415 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.655127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.0.1:41998)
0928 09:07:18.655544 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.655554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.655668 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.655932 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.655933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.655935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:18.655987 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.655989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.657367 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.657368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.658385 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.658387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.659164 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.659165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.659222 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.659225 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.659275 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.659309 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260928 09:07:18.659850 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.655167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.0.1:44120)
0928 09:07:18.655552 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.655571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.655616 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.655849 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.655849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.655851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:18.655913 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.655916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.657646 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.657647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.658714 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.658717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.659598 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.659599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.659662 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.659666 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.659723 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.659768 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":19,"thread_start_us":79,"threads_started":1}
I20260928 09:07:18.660172 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.655394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.0.98.1:37377)
0928 09:07:18.655623 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.655626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.655638 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:07:18.655678 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.655680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.655693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.655762 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:18.656177 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.657263 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.658805 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.659514 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.659588 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.659646 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.660027 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.660081 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260928 09:07:18.660492 16020 tablet_service.cc:1467] Processing CreateTablet for tablet 337b847868e84151aca6ea8468bb4b0a (DEFAULT_TABLE table=test-table [id=f96da489b3534612821259c7cc4aeaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:18.660830 16020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 337b847868e84151aca6ea8468bb4b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:18.660761 15886 tablet_service.cc:1467] Processing CreateTablet for tablet 337b847868e84151aca6ea8468bb4b0a (DEFAULT_TABLE table=test-table [id=f96da489b3534612821259c7cc4aeaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:18.661078 15886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 337b847868e84151aca6ea8468bb4b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:18.662177 16154 tablet_service.cc:1467] Processing CreateTablet for tablet 337b847868e84151aca6ea8468bb4b0a (DEFAULT_TABLE table=test-table [id=f96da489b3534612821259c7cc4aeaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:18.662490 16154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 337b847868e84151aca6ea8468bb4b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:18.663856 16237 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap starting.
I20260928 09:07:18.664196 16238 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Bootstrap starting.
I20260928 09:07:18.664522 16237 tablet_bootstrap.cc:654] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:18.664826 16238 tablet_bootstrap.cc:654] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:18.665053 16237 log.cc:824] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:18.665371 16238 log.cc:824] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:18.665907 16237 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: No bootstrap required, opened a new log
I20260928 09:07:18.665897 16241 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Bootstrap starting.
I20260928 09:07:18.665978 16237 ts_tablet_manager.cc:1397] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:07:18.666237 16238 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: No bootstrap required, opened a new log
I20260928 09:07:18.666360 16238 ts_tablet_manager.cc:1397] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:07:18.666540 16241 tablet_bootstrap.cc:654] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:18.667116 16241 log.cc:824] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:18.667891 16241 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: No bootstrap required, opened a new log
I20260928 09:07:18.667960 16241 ts_tablet_manager.cc:1397] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:07:18.667845 16237 raft_consensus.cc:348] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.668011 16237 raft_consensus.cc:374] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:18.668076 16237 raft_consensus.cc:729] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e5d0eb258464896d8da3789dc3adb, State: Initialized, Role: FOLLOWER
I20260928 09:07:18.668201 16237 consensus_queue.cc:260] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.668480 16237 ts_tablet_manager.cc:1428] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:07:18.668416 16238 raft_consensus.cc:348] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.668589 16238 raft_consensus.cc:374] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:18.668632 16238 raft_consensus.cc:729] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797f41f873a34a978f2dfa0e8f29ef0b, State: Initialized, Role: FOLLOWER
I20260928 09:07:18.668645 15952 heartbeater.cc:503] Master 127.0.98.62:43785 was elected leader, sending a full tablet report...
I20260928 09:07:18.668759 16238 consensus_queue.cc:260] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.669018 16238 ts_tablet_manager.cc:1428] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:07:18.669047 16086 heartbeater.cc:503] Master 127.0.98.62:43785 was elected leader, sending a full tablet report...
I20260928 09:07:18.670298 16241 raft_consensus.cc:348] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.670442 16241 raft_consensus.cc:374] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:18.670487 16241 raft_consensus.cc:729] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30baa5450fe84daeb490c0e31f2f7eb8, State: Initialized, Role: FOLLOWER
I20260928 09:07:18.670603 16241 consensus_queue.cc:260] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.670845 16241 ts_tablet_manager.cc:1428] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 09:07:18.670908 16220 heartbeater.cc:503] Master 127.0.98.62:43785 was elected leader, sending a full tablet report...
W20260928 09:07:18.725019 16087 tablet.cc:2369] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:07:18.816306 15953 tablet.cc:2369] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:07:18.826352 16245 raft_consensus.cc:482] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:18.826478 16245 raft_consensus.cc:504] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.826988 16245 leader_election.cc:288] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377), 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183)
I20260928 09:07:18.831064 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.827230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33589 (local address 127.0.98.2:39183)
0928 09:07:18.827281 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.827283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.827295 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:07:18.827436 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.827437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.827450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.827513 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:18.827701 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.828465 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.830113 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.830710 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.830776 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.830785 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.830950 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.830993 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:07:18.831090 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.827211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.98.3:44681)
0928 09:07:18.827403 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.827417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.827488 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.827700 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.827701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.827703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:18.827764 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.827766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.828608 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.828609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.830012 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.830016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.830784 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.830785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.830866 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.830871 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.830931 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.830978 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":56,"thread_start_us":119,"threads_started":1}
I20260928 09:07:18.831136 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.827182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39183 (local address 127.0.98.3:33589)
0928 09:07:18.827237 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.827251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:18.827304 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:18.827536 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:18.827537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:18.827539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:18.827614 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.827616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.828589 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.828590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.830012 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:18.830016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.830784 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.830785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:18.830869 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.830871 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:07:18.830931 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 09:07:18.830978 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":102}
I20260928 09:07:18.831138 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.827316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44681 (local address 127.0.98.1:37377)
0928 09:07:18.827389 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.827391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.827491 (+ 100us) server_negotiation.cc:407] Connection header received
0928 09:07:18.827609 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.827610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.827624 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.827696 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:18.827831 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.828507 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.830123 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.830671 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.830757 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.830770 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.830950 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.830992 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 09:07:18.831512 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "337b847868e84151aca6ea8468bb4b0a" candidate_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" is_pre_election: true
I20260928 09:07:18.831480 15906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "337b847868e84151aca6ea8468bb4b0a" candidate_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e5d0eb258464896d8da3789dc3adb" is_pre_election: true
I20260928 09:07:18.831679 16040 raft_consensus.cc:2454] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30baa5450fe84daeb490c0e31f2f7eb8 in term 0.
I20260928 09:07:18.831692 15906 raft_consensus.cc:2454] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30baa5450fe84daeb490c0e31f2f7eb8 in term 0.
I20260928 09:07:18.831930 16110 leader_election.cc:302] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b; no voters:
I20260928 09:07:18.832108 16245 raft_consensus.cc:2781] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:07:18.832165 16245 raft_consensus.cc:482] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:07:18.832191 16245 raft_consensus.cc:3037] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:18.832872 16245 raft_consensus.cc:504] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.833024 16245 leader_election.cc:288] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377), 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183)
I20260928 09:07:18.833182 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "337b847868e84151aca6ea8468bb4b0a" candidate_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
I20260928 09:07:18.833200 15906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "337b847868e84151aca6ea8468bb4b0a" candidate_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e5d0eb258464896d8da3789dc3adb"
I20260928 09:07:18.833257 16040 raft_consensus.cc:3037] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:18.833308 15906 raft_consensus.cc:3037] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:18.834045 16040 raft_consensus.cc:2454] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30baa5450fe84daeb490c0e31f2f7eb8 in term 1.
I20260928 09:07:18.834110 15906 raft_consensus.cc:2454] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30baa5450fe84daeb490c0e31f2f7eb8 in term 1.
I20260928 09:07:18.834241 16110 leader_election.cc:302] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b; no voters:
I20260928 09:07:18.834331 16245 raft_consensus.cc:2781] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:07:18.834471 16245 raft_consensus.cc:686] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 30baa5450fe84daeb490c0e31f2f7eb8, State: Running, Role: LEADER
I20260928 09:07:18.834586 16245 consensus_queue.cc:237] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:18.838644 15760 catalog_manager.cc:5495] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: HEALTHY } } }
W20260928 09:07:18.874800 16221 tablet.cc:2369] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:07:18.885103 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.882635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.0.98.3:36065)
0928 09:07:18.882705 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:18.882707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:18.882718 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:07:18.882763 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:18.882764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:18.882785 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:18.882849 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:07:18.883031 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.883527 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.884340 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:18.884548 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:18.884609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:18.884775 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:07:18.884878 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:07:18.884942 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:18.884987 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 09:07:18.885007 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:07:18.887140 16040 raft_consensus.cc:1261] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Refusing update from remote peer 30baa5450fe84daeb490c0e31f2f7eb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:18.887461 15906 raft_consensus.cc:1261] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Refusing update from remote peer 30baa5450fe84daeb490c0e31f2f7eb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:18.887518 16245 consensus_queue.cc:1035] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:18.887809 16245 consensus_queue.cc:1035] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:18.890013 16250 mvcc.cc:205] Tried to move back new op lower bound from 7334242053678481408 to 7334242053467537408. Current Snapshot: MvccSnapshot[applied={T|T < 7334242053678481408}]
I20260928 09:07:18.890378 16251 mvcc.cc:205] Tried to move back new op lower bound from 7334242053678481408 to 7334242053467537408. Current Snapshot: MvccSnapshot[applied={T|T < 7334242053678481408}]
I20260928 09:07:18.890645 16252 mvcc.cc:205] Tried to move back new op lower bound from 7334242053678481408 to 7334242053467537408. Current Snapshot: MvccSnapshot[applied={T|T < 7334242053678481408}]
I20260928 09:07:18.892274 15760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:18.892410 15760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:18.893325 15760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Sending DeleteTablet for 3 replicas of tablet 337b847868e84151aca6ea8468bb4b0a
I20260928 09:07:18.893749 16154 tablet_service.cc:1514] Processing DeleteTablet for tablet 337b847868e84151aca6ea8468bb4b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41998
I20260928 09:07:18.893750 15886 tablet_service.cc:1514] Processing DeleteTablet for tablet 337b847868e84151aca6ea8468bb4b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44120
I20260928 09:07:18.893981 16260 tablet_replica.cc:330] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: stopping tablet replica
I20260928 09:07:18.893967 16020 tablet_service.cc:1514] Processing DeleteTablet for tablet 337b847868e84151aca6ea8468bb4b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44568
I20260928 09:07:18.894101 16260 raft_consensus.cc:2229] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:18.894187 16261 tablet_replica.cc:330] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: stopping tablet replica
I20260928 09:07:18.894192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15823
I20260928 09:07:18.894310 16261 raft_consensus.cc:2229] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:18.894418 16261 raft_consensus.cc:2258] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:18.895025 16261 ts_tablet_manager.cc:1905] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:18.896888 16261 ts_tablet_manager.cc:1918] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:18.896977 16261 log.cc:1196] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/337b847868e84151aca6ea8468bb4b0a
I20260928 09:07:18.897310 16261 ts_tablet_manager.cc:1939] T 337b847868e84151aca6ea8468bb4b0a P 797f41f873a34a978f2dfa0e8f29ef0b: Deleting consensus metadata
I20260928 09:07:18.897663 16259 tablet_replica.cc:330] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: stopping tablet replica
I20260928 09:07:18.897770 16259 raft_consensus.cc:2229] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:07:18.897866 16259 raft_consensus.cc:2258] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:18.898231 16259 ts_tablet_manager.cc:1905] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:18.899703 16259 ts_tablet_manager.cc:1918] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:18.899791 16259 log.cc:1196] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/337b847868e84151aca6ea8468bb4b0a
I20260928 09:07:18.900104 16259 ts_tablet_manager.cc:1939] T 337b847868e84151aca6ea8468bb4b0a P 30baa5450fe84daeb490c0e31f2f7eb8: Deleting consensus metadata
I20260928 09:07:18.900592 15745 catalog_manager.cc:4841] TS 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): tablet 337b847868e84151aca6ea8468bb4b0a (table test-table [id=f96da489b3534612821259c7cc4aeaab]) successfully deleted
W20260928 09:07:18.900662 15746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37377 (error 108)
I20260928 09:07:18.900867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15957
I20260928 09:07:18.900913 15748 catalog_manager.cc:4841] TS 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): tablet 337b847868e84151aca6ea8468bb4b0a (table test-table [id=f96da489b3534612821259c7cc4aeaab]) successfully deleted
I20260928 09:07:18.901268 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:18.901151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.0.1:44134)
0928 09:07:18.901200 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:18.901231 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 09:07:18.901535 15746 catalog_manager.cc:4584] TS 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 337b847868e84151aca6ea8468bb4b0a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37377: connect: Connection refused (error 111)
I20260928 09:07:18.906502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 16091
I20260928 09:07:18.911856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 15729
I20260928 09:07:38.917605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--webserver_interface=127.0.98.62
--webserver_port=35321
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:39.000941 16263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:39.001173 16263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:39.001195 16263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:39.002566 16263 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 09:07:39.002615 16263 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 09:07:39.002629 16263 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 09:07:39.002641 16263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:07:39.002655 16263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:39.002668 16263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:39.002681 16263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:07:39.002692 16263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:07:39.004204 16263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:39.004426 16263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:39.004606 16263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:39.013635 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:39.013607 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:39.013607 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:39.013677 16263 server_base.cc:1034] running on GCE node
I20260928 09:07:39.014242 16263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:39.014751 16263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:39.015975 16263 hybrid_clock.cc:648] HybridClock initialized: now 1790586459015959 us; error 26 us; skew 500 ppm
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:39.027204 16263 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:07:39.027761 16263 webserver.cc:460] Webserver started at http://127.0.98.62:35321/ using document root <none> and password file <none>
I20260928 09:07:39.028046 16263 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:39.028108 16263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:39.029663 16263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:07:39.030483 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.030710 16263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:39.030802 16263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "201b005f2f774cb491b24adc23d9db24"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "583fdac320ad3806afc626096072f0f6"
server_key_iv: "9b0448ff930955fa30de278d24fb1d89"
server_key_version: "encryptionkey@0"
I20260928 09:07:39.031282 16263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:39.074035 16263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:39.076370 16263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:39.076488 16263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:39.080417 16263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43785
I20260928 09:07:39.080497 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43785 every 8 connection(s)
I20260928 09:07:39.080890 16263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:07:39.082463 16332 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:07:39.083144 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap starting.
I20260928 09:07:39.083545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16263
I20260928 09:07:39.083845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37377
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43785
--webserver_port=45607
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:39.092377 16332 log.cc:824] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:39.094646 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:39.094945 16332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap complete.
I20260928 09:07:39.097247 16332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:39.097508 16332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201b005f2f774cb491b24adc23d9db24, State: Initialized, Role: FOLLOWER
I20260928 09:07:39.097733 16332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:39.097826 16332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:07:39.097905 16332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:07:39.097961 16332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:07:39.099097 16332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:39.099220 16332 leader_election.cc:302] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201b005f2f774cb491b24adc23d9db24; no voters:
I20260928 09:07:39.099419 16332 leader_election.cc:288] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:07:39.099457 16336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:07:39.099574 16336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 LEADER]: Becoming Leader. State: Replica: 201b005f2f774cb491b24adc23d9db24, State: Running, Role: LEADER
I20260928 09:07:39.099697 16332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:07:39.099727 16336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:39.100045 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201b005f2f774cb491b24adc23d9db24. Latest consensus state: current_term: 2 leader_uuid: "201b005f2f774cb491b24adc23d9db24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260928 09:07:39.100050 16337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "201b005f2f774cb491b24adc23d9db24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260928 09:07:39.100328 16337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:39.100253 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:39.100566 16345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:07:39.101457 16345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:07:39.101645 16345 catalog_manager.cc:1259] Loaded cluster ID: 706b92f67992436092efac4d3fbb2aa5
I20260928 09:07:39.101879 16345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:07:39.103034 16345 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:07:39.103225 16345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Loaded TSK: 0
I20260928 09:07:39.120121 16345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Generated new TSK 1
I20260928 09:07:39.120237 16345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:07:39.168759 16334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:39.168983 16334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:39.169045 16334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:39.170316 16334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:39.170379 16334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:39.170429 16334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:07:39.171926 16334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45607
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:39.172204 16334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:39.172448 16334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:39.178447 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:39.178540 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:39.178426 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:39.178844 16334 server_base.cc:1034] running on GCE node
I20260928 09:07:39.179050 16334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:39.179307 16334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:39.180482 16334 hybrid_clock.cc:648] HybridClock initialized: now 1790586459180466 us; error 32 us; skew 500 ppm
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:39.183121 16334 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:07:39.183569 16334 webserver.cc:460] Webserver started at http://127.0.98.1:45607/ using document root <none> and password file <none>
I20260928 09:07:39.183809 16334 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:39.183892 16334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:39.185300 16334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:39.186232 16369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.186481 16334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:39.186599 16334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "4b5e5d0eb258464896d8da3789dc3adb"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7c607bfd47158eb699facedb29ac1482"
server_key_iv: "7e02546d09ee080c15ba53e1bf6a512a"
server_key_version: "encryptionkey@0"
I20260928 09:07:39.186960 16334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:39.218904 16334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:39.221271 16334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:39.221442 16334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:39.221736 16334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:39.222215 16376 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:07:39.223779 16334 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:07:39.223836 16334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:07:39.223863 16334 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:07:39.224522 16334 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:07:39.224572 16334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:39.224614 16376 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap starting.
I20260928 09:07:39.232270 16334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37377
I20260928 09:07:39.232342 16483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37377 every 8 connection(s)
I20260928 09:07:39.232867 16334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:07:39.235711 16376 log.cc:824] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:39.237995 16376 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:39.238320 16376 tablet_bootstrap.cc:492] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap complete.
I20260928 09:07:39.239501 16376 ts_tablet_manager.cc:1397] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260928 09:07:39.239648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16334
I20260928 09:07:39.239941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39183
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43785
--webserver_port=36477
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:39.240710 16376 raft_consensus.cc:348] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:39.240991 16376 raft_consensus.cc:729] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e5d0eb258464896d8da3789dc3adb, State: Initialized, Role: FOLLOWER
I20260928 09:07:39.241166 16376 consensus_queue.cc:260] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } }
I20260928 09:07:39.241500 16376 ts_tablet_manager.cc:1428] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:39.249217 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.233728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:49401)
0928 09:07:39.235978 (+ 2250us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.236032 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.238045 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.238563 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.238568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.238977 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:39.239264 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.239270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.240902 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.240903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.241973 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.241976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.242657 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.242659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.242796 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.243434 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:39.243444 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:39.245116 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:39.247599 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.247605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.247606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.247838 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.248130 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.248131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.248131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.248235 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.248737 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:39.248741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:39.248853 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.248932 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.249057 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2203,"thread_start_us":95,"threads_started":1}
I20260928 09:07:39.249433 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.233759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49401 (local address 127.0.98.62:43785)
0928 09:07:39.233935 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.233944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.236181 (+ 2237us) server_negotiation.cc:407] Connection header received
0928 09:07:39.238303 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.238309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.238371 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.238508 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:39.239380 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.240784 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.242102 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.242550 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.242688 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.245229 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:39.245235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:39.245235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:39.245249 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:39.247490 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.247926 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.247930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.247930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.248057 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.248307 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.248308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.248309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.248662 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:39.248757 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.249025 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.249185 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260928 09:07:39.250293 16484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43785
I20260928 09:07:39.250391 16484 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:39.250582 16484 heartbeater.cc:511] Master 127.0.98.62:43785 requested a full tablet report, sending...
I20260928 09:07:39.251351 16296 ts_manager.cc:194] Registered new tserver with Master: 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377)
I20260928 09:07:39.252693 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49401
I20260928 09:07:39.253670 16484 heartbeater.cc:503] Master 127.0.98.62:43785 was elected leader, sending a full tablet report...
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:07:39.260617 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.252047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.0.1:38830)
0928 09:07:39.252333 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.252366 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.252459 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.252671 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.252676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.253043 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:39.253134 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.253137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.254494 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.254495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.255097 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.255099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.255712 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.255715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.255779 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.256282 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:39.256287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:39.257627 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:39.259566 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.259573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.259574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.259794 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.259997 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.259998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.259998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.260054 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.260344 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:39.260349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:39.260393 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.260456 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.260547 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":155,"threads_started":1}
I20260928 09:07:39.260708 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.252097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.0.98.1:37377)
0928 09:07:39.252287 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.252293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.252441 (+ 148us) server_negotiation.cc:407] Connection header received
0928 09:07:39.252556 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.252559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.252579 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.252656 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:39.253198 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.254340 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.255182 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.255640 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.255708 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.257731 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:39.257739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:39.257739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:39.257746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:39.259477 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.259870 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.259873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.259873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.259932 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.260112 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.260113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.260113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.260283 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:39.260359 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.260508 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.260611 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260928 09:07:39.261260 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet 337b847868e84151aca6ea8468bb4b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38830
I20260928 09:07:39.261914 16493 tablet_replica.cc:330] stopping tablet replica
I20260928 09:07:39.262022 16493 raft_consensus.cc:2229] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:39.262094 16493 raft_consensus.cc:2258] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:39.262378 16493 ts_tablet_manager.cc:1905] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:39.264791 16493 ts_tablet_manager.cc:1918] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:39.264851 16493 log.cc:1196] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal/wals/337b847868e84151aca6ea8468bb4b0a
I20260928 09:07:39.265087 16493 ts_tablet_manager.cc:1939] T 337b847868e84151aca6ea8468bb4b0a P 4b5e5d0eb258464896d8da3789dc3adb: Deleting consensus metadata
I20260928 09:07:39.265604 16281 catalog_manager.cc:4841] TS 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377): tablet 337b847868e84151aca6ea8468bb4b0a (table test-table [id=f96da489b3534612821259c7cc4aeaab]) successfully deleted
W20260928 09:07:39.327646 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:39.327890 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:39.327930 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:39.329121 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:39.329188 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:39.329233 16489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:07:39.330730 16489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36477
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:39.330999 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:39.331239 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:39.336308 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:39.336365 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:39.336494 16489 server_base.cc:1034] running on GCE node
W20260928 09:07:39.336323 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:39.336838 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:39.337093 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:39.338255 16489 hybrid_clock.cc:648] HybridClock initialized: now 1790586459338242 us; error 31 us; skew 500 ppm
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:39.340768 16489 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:07:39.341150 16489 webserver.cc:460] Webserver started at http://127.0.98.2:36477/ using document root <none> and password file <none>
I20260928 09:07:39.341413 16489 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:39.341516 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:39.342950 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:07:39.343542 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.343714 16489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:07:39.343792 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7a3490f4dc0147f135ecbaa3767085ce"
server_key_iv: "18467daf9fd6e4fc34be8c49a6ad8650"
server_key_version: "encryptionkey@0"
I20260928 09:07:39.344118 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:39.359122 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:39.360976 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:39.361096 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:39.361291 16489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:39.361655 16489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:39.361692 16489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.361716 16489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:39.361757 16489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.367549 16489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39183
I20260928 09:07:39.367626 16621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39183 every 8 connection(s)
I20260928 09:07:39.367970 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:39.376219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16489
I20260928 09:07:39.376504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36065
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43785
--webserver_port=41411
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:39.377763 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.368746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45869 (local address 127.0.98.62:43785)
0928 09:07:39.368816 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.368818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.369021 (+ 203us) server_negotiation.cc:407] Connection header received
0928 09:07:39.369603 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.369605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.369621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.369685 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:39.370643 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.371153 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.372184 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.372412 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.372490 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.374789 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:39.374791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:39.374791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:39.374798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:39.376181 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.376725 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.376726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.376727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.376790 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.377016 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.377018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.377018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.377178 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:39.377263 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.377498 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.377621 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:07:39.377832 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.368733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:45869)
0928 09:07:39.368897 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.368933 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.369452 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.369773 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.369776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.370340 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:39.370534 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.370539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.371253 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.371255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.372081 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.372083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.372497 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.372498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.372629 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.373207 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:39.373216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:39.374697 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:39.376351 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.376358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.376361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.376627 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.376853 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.376853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.376854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.376945 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.377252 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:39.377256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:39.377372 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.377452 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.377651 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260928 09:07:39.378495 16622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43785
I20260928 09:07:39.378609 16622 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:39.378806 16622 heartbeater.cc:511] Master 127.0.98.62:43785 requested a full tablet report, sending...
I20260928 09:07:39.379204 16296 ts_manager.cc:194] Registered new tserver with Master: 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183)
I20260928 09:07:39.379575 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45869
W20260928 09:07:39.461320 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:39.461516 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:39.461580 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:39.462744 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:39.462785 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:39.462811 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:07:39.464219 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41411
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:39.464442 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:39.464674 16625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:39.469571 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:39.469691 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:39.469666 16625 server_base.cc:1034] running on GCE node
W20260928 09:07:39.469607 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:39.470132 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:39.470345 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:39.471532 16625 hybrid_clock.cc:648] HybridClock initialized: now 1790586459471518 us; error 45 us; skew 500 ppm
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:39.474102 16625 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:07:39.474535 16625 webserver.cc:460] Webserver started at http://127.0.98.3:41411/ using document root <none> and password file <none>
I20260928 09:07:39.474769 16625 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:39.474850 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:39.476178 16625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:39.476747 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.476910 16625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.477002 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "30baa5450fe84daeb490c0e31f2f7eb8"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "9755dc11748db9c260b2034ea542b6ab"
server_key_iv: "fe86b677756f1955e0f9f48e16879ef9"
server_key_version: "encryptionkey@0"
I20260928 09:07:39.477348 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:39.494114 16625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:39.496261 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:39.496456 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:39.496709 16625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:39.497026 16625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:07:39.497082 16625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.497124 16625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:07:39.497152 16625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.503136 16625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36065
I20260928 09:07:39.503216 16753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36065 every 8 connection(s)
I20260928 09:07:39.503578 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:07:39 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:07:39.512537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16625
I20260928 09:07:39.513170 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.504380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59315 (local address 127.0.98.62:43785)
0928 09:07:39.504480 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.504483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.504685 (+ 202us) server_negotiation.cc:407] Connection header received
0928 09:07:39.505250 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.505251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.505264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.505319 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:39.506309 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.506815 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.507845 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.508055 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.508132 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.510356 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:39.510358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:39.510358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:39.510366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:39.511626 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.512108 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.512109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.512109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.512158 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.512446 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.512448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.512448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.512569 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:39.512675 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.512990 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.513081 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:07:39.513372 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.504344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:59315)
0928 09:07:39.504561 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.504597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.505134 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.505396 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.505405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.506011 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:07:39.506204 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.506208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.506927 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.506928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.507737 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.507739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.508136 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.508137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.508263 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.508794 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:07:39.508802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:07:39.510261 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:07:39.511718 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.511724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.511726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.511949 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.512228 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:07:39.512229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:07:39.512229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:07:39.512355 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:07:39.512681 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:07:39.512687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:07:39.512854 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.512967 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.513123 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260928 09:07:39.513868 16754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43785
I20260928 09:07:39.513962 16754 heartbeater.cc:464] Registering TS with master...
I20260928 09:07:39.514159 16754 heartbeater.cc:511] Master 127.0.98.62:43785 requested a full tablet report, sending...
I20260928 09:07:39.514547 16296 ts_manager.cc:194] Registered new tserver with Master: 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
I20260928 09:07:39.514918 16296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59315
I20260928 09:07:39.519498 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.512935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.0.98.62:43785)
0928 09:07:39.513057 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.513059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.513209 (+ 150us) server_negotiation.cc:407] Connection header received
0928 09:07:39.513297 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.513298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.513310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.513360 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:39.514217 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.514758 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.515578 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.515772 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.515874 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.517093 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:39.517095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:39.517095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:39.517101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:39.518366 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.518709 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.518710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.518710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.518771 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.518978 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.518979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.518979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.519089 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:39.519268 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.519309 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.519391 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260928 09:07:39.520193 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:07:39.523075 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.520526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.0.98.62:43785)
0928 09:07:39.520589 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.520590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.520683 (+ 93us) server_negotiation.cc:407] Connection header received
0928 09:07:39.520771 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.520772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.520808 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.520890 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:07:39.521010 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.521496 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.522305 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.522520 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.522605 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.522775 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:07:39.522830 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 09:07:39.523011 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45210: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 09:07:39.523152 16758 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45210: token expired
W20260928 09:07:39.523207 16226 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43785)
I20260928 09:07:39.529306 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.523413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.0.98.62:43785)
0928 09:07:39.523469 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.523471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.523503 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:07:39.523572 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.523573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.523583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.523636 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:07:39.524106 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.524573 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.525290 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.525493 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.525584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.526801 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:07:39.526803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:07:39.526803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:07:39.526810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:07:39.528154 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.528522 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.528523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.528523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.528584 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:07:39.528790 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:07:39.528791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:07:39.528791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:07:39.528905 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:07:39.529007 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.529126 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.529204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:07:39.530684 16296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:07:39.532333 16296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:07:39.534835 16400 tablet_service.cc:1467] Processing CreateTablet for tablet f484ff5e51a5479784ea2a7931ff4f40 (DEFAULT_TABLE table=test-table [id=ba3e8f236b2948f1bdb452ff5eaf962c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:39.535053 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f484ff5e51a5479784ea2a7931ff4f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:39.536329 16376 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap starting.
I20260928 09:07:39.536659 16376 tablet_bootstrap.cc:654] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:39.537484 16376 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: No bootstrap required, opened a new log
I20260928 09:07:39.537547 16376 ts_tablet_manager.cc:1397] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:07:39.537731 16376 raft_consensus.cc:348] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.537782 16376 raft_consensus.cc:374] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:39.537813 16376 raft_consensus.cc:729] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e5d0eb258464896d8da3789dc3adb, State: Initialized, Role: FOLLOWER
I20260928 09:07:39.537874 16376 consensus_queue.cc:260] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.537964 16376 ts_tablet_manager.cc:1428] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:39.539072 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.534528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39183 (local address 127.0.0.1:36110)
0928 09:07:39.534580 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.534589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.534630 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.535198 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.535199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.535201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:39.535263 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.535266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.536574 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.536575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.537864 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.537867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.538772 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.538773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.538851 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.538874 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.538939 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.538990 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":44}
I20260928 09:07:39.539090 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.534639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.98.2:39183)
0928 09:07:39.535037 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.535041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.535050 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:07:39.535083 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.535084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.535097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.535193 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:39.535349 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.536468 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.537950 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.538666 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.538767 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.538837 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.538955 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.538997 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":92,"threads_started":1}
I20260928 09:07:39.539230 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.534602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.0.1:35364)
0928 09:07:39.534846 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.534858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.534912 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.535346 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.535347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.535349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:39.535398 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.535400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.536619 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.536620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.537864 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.537867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.538952 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.538953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.539036 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.539040 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.539109 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.539156 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":6,"thread_start_us":80,"threads_started":1}
I20260928 09:07:39.539292 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.534815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.0.98.3:36065)
0928 09:07:39.534994 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.534999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.535011 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:07:39.535053 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.535055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.535156 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.535244 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:39.535482 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.536536 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.537968 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.538876 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.538947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.539002 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.539135 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.539183 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260928 09:07:39.539968 16688 tablet_service.cc:1467] Processing CreateTablet for tablet f484ff5e51a5479784ea2a7931ff4f40 (DEFAULT_TABLE table=test-table [id=ba3e8f236b2948f1bdb452ff5eaf962c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:39.540086 16556 tablet_service.cc:1467] Processing CreateTablet for tablet f484ff5e51a5479784ea2a7931ff4f40 (DEFAULT_TABLE table=test-table [id=ba3e8f236b2948f1bdb452ff5eaf962c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:07:39.540339 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f484ff5e51a5479784ea2a7931ff4f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:39.540442 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f484ff5e51a5479784ea2a7931ff4f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:07:39.543424 16764 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Bootstrap starting.
I20260928 09:07:39.543458 16765 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Bootstrap starting.
I20260928 09:07:39.544138 16764 tablet_bootstrap.cc:654] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:39.544142 16765 tablet_bootstrap.cc:654] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Neither blocks nor log segments found. Creating new log.
I20260928 09:07:39.544634 16765 log.cc:824] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:39.544646 16764 log.cc:824] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:39.545286 16765 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: No bootstrap required, opened a new log
I20260928 09:07:39.545296 16764 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: No bootstrap required, opened a new log
I20260928 09:07:39.545365 16765 ts_tablet_manager.cc:1397] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:07:39.545357 16764 ts_tablet_manager.cc:1397] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:07:39.546944 16764 raft_consensus.cc:348] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.546916 16765 raft_consensus.cc:348] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.547138 16765 raft_consensus.cc:374] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:39.547139 16764 raft_consensus.cc:374] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:07:39.547185 16764 raft_consensus.cc:729] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30baa5450fe84daeb490c0e31f2f7eb8, State: Initialized, Role: FOLLOWER
I20260928 09:07:39.547178 16765 raft_consensus.cc:729] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797f41f873a34a978f2dfa0e8f29ef0b, State: Initialized, Role: FOLLOWER
I20260928 09:07:39.547318 16765 consensus_queue.cc:260] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.547319 16764 consensus_queue.cc:260] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.547619 16764 ts_tablet_manager.cc:1428] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:07:39.547627 16765 ts_tablet_manager.cc:1428] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:07:39.547747 16754 heartbeater.cc:503] Master 127.0.98.62:43785 was elected leader, sending a full tablet report...
I20260928 09:07:39.547744 16622 heartbeater.cc:503] Master 127.0.98.62:43785 was elected leader, sending a full tablet report...
I20260928 09:07:39.618281 16490 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:07:39.618458 16490 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
W20260928 09:07:39.618537 16623 tablet.cc:2369] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:07:39.618881 16490 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
I20260928 09:07:39.621989 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.619194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38967 (local address 127.0.98.3:36065)
0928 09:07:39.619256 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.619258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.619268 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:07:39.619346 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.619347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.619360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.619451 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:39.619618 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.620114 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.621170 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.621597 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.621693 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.621703 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.621879 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.621916 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 09:07:39.621989 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.619061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.98.1:38967)
0928 09:07:39.619168 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.619181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.619243 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.619457 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.619458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.619460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:39.619539 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.619540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.620196 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.620197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.621068 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.621071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.621693 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.621694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.621775 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.621794 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.621872 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.621917 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":64}
I20260928 09:07:39.622068 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.619116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59623 (local address 127.0.98.2:39183)
0928 09:07:39.619165 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.619168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.619380 (+ 212us) server_negotiation.cc:407] Connection header received
0928 09:07:39.619450 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.619452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.619465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.619541 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:07:39.619712 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.620167 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.621250 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.621693 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.621780 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.621791 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.621962 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.621997 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:07:39.622097 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.619086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39183 (local address 127.0.98.1:59623)
0928 09:07:39.619306 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.619314 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:39.619395 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:07:39.619546 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:07:39.619547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:07:39.619548 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:07:39.619641 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.619643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.620293 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.620294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.621148 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:07:39.621151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.621782 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.621783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:07:39.621876 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.621879 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:07:39.621954 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 09:07:39.621994 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260928 09:07:39.622459 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f484ff5e51a5479784ea2a7931ff4f40" candidate_uuid: "4b5e5d0eb258464896d8da3789dc3adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" is_pre_election: true
I20260928 09:07:39.622527 16708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f484ff5e51a5479784ea2a7931ff4f40" candidate_uuid: "4b5e5d0eb258464896d8da3789dc3adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" is_pre_election: true
I20260928 09:07:39.622619 16576 raft_consensus.cc:2454] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5e5d0eb258464896d8da3789dc3adb in term 0.
I20260928 09:07:39.622681 16708 raft_consensus.cc:2454] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5e5d0eb258464896d8da3789dc3adb in term 0.
I20260928 09:07:39.622834 16373 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb, 797f41f873a34a978f2dfa0e8f29ef0b; no voters:
I20260928 09:07:39.622983 16490 raft_consensus.cc:2781] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:07:39.623046 16490 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:07:39.623070 16490 raft_consensus.cc:3037] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:39.623749 16490 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.623883 16490 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 1 election: Requested vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
I20260928 09:07:39.624019 16708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f484ff5e51a5479784ea2a7931ff4f40" candidate_uuid: "4b5e5d0eb258464896d8da3789dc3adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30baa5450fe84daeb490c0e31f2f7eb8"
I20260928 09:07:39.624032 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f484ff5e51a5479784ea2a7931ff4f40" candidate_uuid: "4b5e5d0eb258464896d8da3789dc3adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
I20260928 09:07:39.624100 16576 raft_consensus.cc:3037] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:39.624100 16708 raft_consensus.cc:3037] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:07:39.624809 16708 raft_consensus.cc:2454] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5e5d0eb258464896d8da3789dc3adb in term 1.
I20260928 09:07:39.624842 16576 raft_consensus.cc:2454] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5e5d0eb258464896d8da3789dc3adb in term 1.
I20260928 09:07:39.625090 16373 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb, 797f41f873a34a978f2dfa0e8f29ef0b; no voters:
I20260928 09:07:39.625284 16490 raft_consensus.cc:2781] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:07:39.625420 16490 raft_consensus.cc:686] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 LEADER]: Becoming Leader. State: Replica: 4b5e5d0eb258464896d8da3789dc3adb, State: Running, Role: LEADER
I20260928 09:07:39.625514 16490 consensus_queue.cc:237] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:39.626448 16296 catalog_manager.cc:5495] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4b5e5d0eb258464896d8da3789dc3adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } health_report { overall_health: HEALTHY } } }
I20260928 09:07:39.631788 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.629286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.98.1:37377)
0928 09:07:39.629334 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:07:39.629336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:07:39.629345 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:07:39.629419 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:07:39.629420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:07:39.629440 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:07:39.629543 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:07:39.629719 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.630214 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.630963 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:07:39.631190 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:07:39.631282 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:07:39.631451 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:07:39.631561 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:07:39.631644 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 09:07:39.631685 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 09:07:39.631706 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:07:39.633270 16708 raft_consensus.cc:1261] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Refusing update from remote peer 4b5e5d0eb258464896d8da3789dc3adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:39.633275 16576 raft_consensus.cc:1261] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Refusing update from remote peer 4b5e5d0eb258464896d8da3789dc3adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:07:39.633663 16490 consensus_queue.cc:1035] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:39.633813 16771 consensus_queue.cc:1035] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:07:39.636461 16774 mvcc.cc:205] Tried to move back new op lower bound from 7334242138655416320 to 7334242138627031040. Current Snapshot: MvccSnapshot[applied={T|T < 7334242138655416320}]
I20260928 09:07:39.636590 16775 mvcc.cc:205] Tried to move back new op lower bound from 7334242138655416320 to 7334242138627031040. Current Snapshot: MvccSnapshot[applied={T|T < 7334242138655416320}]
I20260928 09:07:39.638336 16295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:39.638435 16295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:07:39.639205 16295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Sending DeleteTablet for 3 replicas of tablet f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:07:39.639511 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet f484ff5e51a5479784ea2a7931ff4f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38830
I20260928 09:07:39.639547 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet f484ff5e51a5479784ea2a7931ff4f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35364
I20260928 09:07:39.639524 16556 tablet_service.cc:1514] Processing DeleteTablet for tablet f484ff5e51a5479784ea2a7931ff4f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:07:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36110
I20260928 09:07:39.639618 16493 tablet_replica.cc:330] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: stopping tablet replica
I20260928 09:07:39.639679 16493 raft_consensus.cc:2229] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:07:39.639766 16493 raft_consensus.cc:2258] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:39.639766 16784 tablet_replica.cc:330] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: stopping tablet replica
I20260928 09:07:39.639818 16785 tablet_replica.cc:330] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: stopping tablet replica
I20260928 09:07:39.639849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 16334
I20260928 09:07:39.639914 16785 raft_consensus.cc:2229] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:39.640017 16784 raft_consensus.cc:2229] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:07:39.640167 16784 raft_consensus.cc:2258] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:39.640586 16785 raft_consensus.cc:2258] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:07:39.640830 16784 ts_tablet_manager.cc:1905] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:39.640951 16785 ts_tablet_manager.cc:1905] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:07:39.642797 16785 ts_tablet_manager.cc:1918] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:39.642876 16785 log.cc:1196] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:07:39.643060 16784 ts_tablet_manager.cc:1918] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:07:39.643182 16784 log.cc:1196] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:07:39.643275 16785 ts_tablet_manager.cc:1939] T f484ff5e51a5479784ea2a7931ff4f40 P 797f41f873a34a978f2dfa0e8f29ef0b: Deleting consensus metadata
I20260928 09:07:39.643628 16283 catalog_manager.cc:4841] TS 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): tablet f484ff5e51a5479784ea2a7931ff4f40 (table test-table [id=ba3e8f236b2948f1bdb452ff5eaf962c]) successfully deleted
I20260928 09:07:39.643771 16784 ts_tablet_manager.cc:1939] T f484ff5e51a5479784ea2a7931ff4f40 P 30baa5450fe84daeb490c0e31f2f7eb8: Deleting consensus metadata
I20260928 09:07:39.644232 16280 catalog_manager.cc:4841] TS 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): tablet f484ff5e51a5479784ea2a7931ff4f40 (table test-table [id=ba3e8f236b2948f1bdb452ff5eaf962c]) successfully deleted
W20260928 09:07:39.646834 16281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37377 (error 108)
I20260928 09:07:39.647082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 16489
I20260928 09:07:39.647840 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:07:39.647252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.0.1:38844)
0928 09:07:39.647761 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:39.647800 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489}
W20260928 09:07:39.648881 16281 catalog_manager.cc:4584] TS 4b5e5d0eb258464896d8da3789dc3adb (127.0.98.1:37377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f484ff5e51a5479784ea2a7931ff4f40: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37377: connect: Connection refused (error 111)
I20260928 09:07:39.652272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 16625
I20260928 09:07:39.656787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 16263
I20260928 09:07:59.662184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--webserver_interface=127.0.98.62
--webserver_port=35321
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:07:59.744239 16787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:59.744451 16787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:59.744473 16787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:59.745705 16787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 09:07:59.745749 16787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 09:07:59.745762 16787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 09:07:59.745774 16787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:07:59.745787 16787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:59.745800 16787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:59.745812 16787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:07:59.745859 16787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:07:59.747431 16787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:59.747689 16787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:59.747938 16787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:07:59.753176 16787 server_base.cc:1034] running on GCE node
W20260928 09:07:59.753103 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:59.753125 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:59.753176 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:59.753701 16787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:59.753988 16787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:59.755162 16787 hybrid_clock.cc:648] HybridClock initialized: now 1790586479755186 us; error 64 us; skew 500 ppm
Sep 28 09:07:59 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:59.764169 16787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:07:59.764684 16787 webserver.cc:460] Webserver started at http://127.0.98.62:35321/ using document root <none> and password file <none>
I20260928 09:07:59.764910 16787 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:59.764955 16787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:59.766419 16787 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:59.767082 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:59.767292 16787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:59.767402 16787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "201b005f2f774cb491b24adc23d9db24"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "583fdac320ad3806afc626096072f0f6"
server_key_iv: "9b0448ff930955fa30de278d24fb1d89"
server_key_version: "encryptionkey@0"
I20260928 09:07:59.767765 16787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:59.780401 16787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:59.782756 16787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:59.782928 16787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:59.786909 16787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43785
I20260928 09:07:59.786943 16854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43785 every 8 connection(s)
I20260928 09:07:59.787334 16787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:07:59.788825 16855 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:07:59.789413 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap starting.
I20260928 09:07:59.797585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16787
I20260928 09:07:59.797724 16855 log.cc:824] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:59.797955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37377
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43785
--webserver_port=45607
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:59.800240 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:59.800555 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Bootstrap complete.
I20260928 09:07:59.804514 16855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:59.804898 16855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201b005f2f774cb491b24adc23d9db24, State: Initialized, Role: FOLLOWER
I20260928 09:07:59.805109 16855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:59.805198 16855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:07:59.805258 16855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:07:59.805315 16855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 2 FOLLOWER]: Advancing to term 3
I20260928 09:07:59.806944 16855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:59.807097 16855 leader_election.cc:302] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201b005f2f774cb491b24adc23d9db24; no voters:
I20260928 09:07:59.807307 16855 leader_election.cc:288] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 09:07:59.807379 16859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 09:07:59.807562 16859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [term 3 LEADER]: Becoming Leader. State: Replica: 201b005f2f774cb491b24adc23d9db24, State: Running, Role: LEADER
I20260928 09:07:59.807583 16855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:07:59.807660 16859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } }
I20260928 09:07:59.807945 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "201b005f2f774cb491b24adc23d9db24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260928 09:07:59.807945 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201b005f2f774cb491b24adc23d9db24. Latest consensus state: current_term: 3 leader_uuid: "201b005f2f774cb491b24adc23d9db24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b005f2f774cb491b24adc23d9db24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43785 } } }
I20260928 09:07:59.808117 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:59.808123 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24 [sys.catalog]: This master's current role is: LEADER
I20260928 09:07:59.808426 16868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:07:59.809278 16868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:07:59.809432 16868 catalog_manager.cc:1259] Loaded cluster ID: 706b92f67992436092efac4d3fbb2aa5
I20260928 09:07:59.809634 16868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:07:59.810771 16868 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:07:59.810935 16868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Loaded TSK: 0
I20260928 09:07:59.810971 16868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 201b005f2f774cb491b24adc23d9db24: Loaded TSK: 1
F20260928 09:07:59.811188 16868 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8136d699ad google::LogMessage::Fail() at ??:0
@ 0x7f8136d6c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8136d694de google::LogMessage::Flush() at ??:0
@ 0x7f8136d6aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f81394d6104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f81394d3d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f813ecc31cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f813ecc348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f813ecc3594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f813ec89c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f813ec93ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8138c370d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8138c2c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8139a906db start_thread at ??:0
@ 0x7f813586761f clone at ??:0
W20260928 09:07:59.908862 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:07:59.909158 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:07:59.909200 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:07:59.911072 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:07:59.911135 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:07:59.911177 16858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:07:59.913347 16858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45607
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:07:59.913637 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:07:59.913902 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:07:59.919416 16897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:59.919591 16858 server_base.cc:1034] running on GCE node
W20260928 09:07:59.919404 16899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:07:59.919401 16896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:07:59.919854 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:07:59.920140 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:07:59.921309 16858 hybrid_clock.cc:648] HybridClock initialized: now 1790586479921291 us; error 35 us; skew 500 ppm
Sep 28 09:07:59 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:07:59.924286 16858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:07:59.924785 16858 webserver.cc:460] Webserver started at http://127.0.98.1:45607/ using document root <none> and password file <none>
I20260928 09:07:59.925012 16858 fs_manager.cc:359] Metadata directory not provided
I20260928 09:07:59.925096 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:07:59.926560 16858 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:59.927325 16906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:07:59.927517 16858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:07:59.927625 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "4b5e5d0eb258464896d8da3789dc3adb"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7c607bfd47158eb699facedb29ac1482"
server_key_iv: "7e02546d09ee080c15ba53e1bf6a512a"
server_key_version: "encryptionkey@0"
I20260928 09:07:59.927973 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:07:59.942414 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:07:59.944722 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:07:59.944888 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:07:59.945133 16858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:07:59.945614 16913 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:07:59.947607 16858 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:07:59.947698 16858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:07:59.947744 16858 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:07:59.948416 16858 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:07:59.948473 16858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 09:07:59.948550 16913 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap starting.
I20260928 09:07:59.955996 16858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37377
I20260928 09:07:59.956128 17020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37377 every 8 connection(s)
I20260928 09:07:59.956436 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:07:59.959753 16913 log.cc:824] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:07:59.962584 16913 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:07:59.962911 16913 tablet_bootstrap.cc:492] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Bootstrap complete.
I20260928 09:07:59.963833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 16858
I20260928 09:07:59.964129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39183
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43785
--webserver_port=36477
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:07:59.964138 16913 ts_tablet_manager.cc:1397] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260928 09:07:59.965613 16913 raft_consensus.cc:348] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:59.965956 16913 raft_consensus.cc:729] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e5d0eb258464896d8da3789dc3adb, State: Initialized, Role: FOLLOWER
I20260928 09:07:59.966136 16913 consensus_queue.cc:260] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:07:59.966421 16913 ts_tablet_manager.cc:1428] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 09:08:00.045909 17023 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:07:59.957071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:44531)
0928 09:07:59.957242 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 09:07:59.957286 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:07:59.957933 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:00.045698 (+ 87765us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
W20260928 09:08:00.046146 17021 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43785 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43785: Client connection negotiation failed: client connection to 127.0.98.62:43785: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:08:00.046479 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.046354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:60089)
0928 09:08:00.046411 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.046450 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:08:00.046823 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.046735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:60853)
0928 09:08:00.046782 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.046805 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 09:08:00.046952 17021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:08:00.060696 17025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:00.060911 17025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:00.060936 17025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:00.062176 17025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:00.062220 17025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:00.062248 17025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:00.063704 17025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36477
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:00.063897 17025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:00.064075 17025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:00.069404 17032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:00.069391 17025 server_base.cc:1034] running on GCE node
W20260928 09:08:00.069312 17031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:00.069314 17034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:00.069762 17025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:00.070030 17025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:00.071195 17025 hybrid_clock.cc:648] HybridClock initialized: now 1790586480071184 us; error 48 us; skew 500 ppm
Sep 28 09:08:00 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:00.073722 17025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:08:00.074198 17025 webserver.cc:460] Webserver started at http://127.0.98.2:36477/ using document root <none> and password file <none>
I20260928 09:08:00.074468 17025 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:00.074538 17025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:00.075846 17025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:00.076498 17041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.076670 17025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.076766 17025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "797f41f873a34a978f2dfa0e8f29ef0b"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "7a3490f4dc0147f135ecbaa3767085ce"
server_key_iv: "18467daf9fd6e4fc34be8c49a6ad8650"
server_key_version: "encryptionkey@0"
I20260928 09:08:00.077121 17025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:00.096104 17025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:00.098165 17025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:00.098338 17025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:00.098583 17025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:00.098891 17025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:00.098945 17025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.098989 17025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:00.099014 17025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.105073 17025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39183
I20260928 09:08:00.105124 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39183 every 8 connection(s)
I20260928 09:08:00.105505 17025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:08:00.106637 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.106216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:60699)
0928 09:08:00.106402 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.106481 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
W20260928 09:08:00.107187 17155 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43785 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43785: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
I20260928 09:08:00.107501 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.107405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:58035)
0928 09:08:00.107452 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.107482 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 09:08:00.107849 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.107735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:42253)
0928 09:08:00.107797 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.107827 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 09:08:00.107997 17155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:08:00.110512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17025
I20260928 09:08:00.110803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36065
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43785
--webserver_port=41411
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:00.192943 17158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:00.193208 17158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:00.193249 17158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:00.194509 17158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:00.194595 17158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:00.194644 17158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:00.196203 17158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41411
--tserver_master_addrs=127.0.98.62:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:00.196480 17158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:00.196727 17158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:00.202018 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:00.202041 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:00.202023 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:00.202270 17158 server_base.cc:1034] running on GCE node
I20260928 09:08:00.202489 17158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:00.202723 17158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:00.203869 17158 hybrid_clock.cc:648] HybridClock initialized: now 1790586480203851 us; error 41 us; skew 500 ppm
Sep 28 09:08:00 dist-test-slave-smns krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:00.206648 17158 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:08:00.207141 17158 webserver.cc:460] Webserver started at http://127.0.98.3:41411/ using document root <none> and password file <none>
I20260928 09:08:00.207371 17158 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:00.207419 17158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:00.208853 17158 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:00.209468 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.209720 17158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:00.209801 17158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "30baa5450fe84daeb490c0e31f2f7eb8"
format_stamp: "Formatted at 2026-09-28 09:07:18 on dist-test-slave-smns"
server_key: "9755dc11748db9c260b2034ea542b6ab"
server_key_iv: "fe86b677756f1955e0f9f48e16879ef9"
server_key_version: "encryptionkey@0"
I20260928 09:08:00.210134 17158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:00.226058 17158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:00.228423 17158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:00.228551 17158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:00.228749 17158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:00.229001 17158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:00.229038 17158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.229060 17158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:00.229072 17158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:00.234798 17158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36065
I20260928 09:08:00.234892 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36065 every 8 connection(s)
I20260928 09:08:00.235229 17158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:08:00.236210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17158
I20260928 09:08:00.236388 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.235898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:60915)
0928 09:08:00.236071 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.236170 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260928 09:08:00.236850 17287 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43785 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43785: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
I20260928 09:08:00.237121 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.237026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:41387)
0928 09:08:00.237074 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.237099 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 09:08:00.237396 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:00.237312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:59211)
0928 09:08:00.237354 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:00.237376 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 09:08:00.237502 17287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:08:01.047631 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.047370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:35777)
0928 09:08:01.047529 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:01.047571 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260928 09:08:01.108587 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.108364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:39137)
0928 09:08:01.108493 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:01.108532 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 09:08:01.238101 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.237802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:55035)
0928 09:08:01.237989 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:01.238035 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260928 09:08:01.565696 17294 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:01.565866 17294 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:08:01.566255 17294 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
Sep 28 09:08:01 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 09:08:01 dist-test-slave-smns krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 09:08:01.577309 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.566417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39183 (local address 127.0.98.1:48367)
0928 09:08:01.566531 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:01.566543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:01.566607 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:01.567514 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:01.567515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:01.567865 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:01.568055 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:01.568061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.569320 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.569321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:01.570147 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:01.570150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.570792 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.570794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:01.570918 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:01.571553 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:01.571557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:01.573298 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:01.575503 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:01.575510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:01.575512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:01.575820 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:01.576118 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:01.576118 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:01.576118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:01.576260 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:01.576742 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:01.576748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:01.576879 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 09:08:01.576987 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 09:08:01.577186 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":51,"thread_start_us":52,"threads_started":1}
I20260928 09:08:01.577380 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.566533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48367 (local address 127.0.98.2:39183)
0928 09:08:01.566680 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:01.566692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:01.566729 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:08:01.567229 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:01.567231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:01.567273 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:01.567428 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:01.568339 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.569195 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.570263 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.570672 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.570812 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:01.573419 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:01.573423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:01.573424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:01.573438 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:01.575416 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:01.575908 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:01.575911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:01.575911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:01.576042 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:01.576348 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:01.576349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:01.576349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:01.576673 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:01.576740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:01.577058 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 09:08:01.577260 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":48,"threads_started":1}
I20260928 09:08:01.577608 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.566591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47177 (local address 127.0.98.3:36065)
0928 09:08:01.566758 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:01.566766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:01.566792 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:08:01.567225 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:01.567228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:01.567273 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:01.567416 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:01.568232 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.569186 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.570243 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.570704 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.571215 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:01.573964 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:01.573969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:01.573969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:01.573983 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:01.575915 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:01.576280 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:01.576283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:01.576283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:01.576384 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:01.576589 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:01.576590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:01.576590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:01.576877 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:01.576961 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:01.577225 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 09:08:01.577358 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260928 09:08:01.577410 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:01.566432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36065 (local address 127.0.98.1:47177)
0928 09:08:01.566558 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:01.566569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:01.566626 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:01.567463 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:01.567465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:01.567864 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:01.568055 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:01.568061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.569297 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.569298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:01.570148 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:01.570151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:01.570829 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:01.570830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:01.570918 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:01.571492 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:01.571518 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:01.573864 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:01.575989 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:01.575990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:01.575990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:01.576199 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:01.576459 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:01.576460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:01.576460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:01.576521 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:01.576953 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:01.576954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:01.577019 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 09:08:01.577096 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 09:08:01.577273 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":84,"thread_start_us":55,"threads_started":1}
W20260928 09:08:01.578730 16910 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
W20260928 09:08:01.578828 16907 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:08:01.578893 16910 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb; no voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b
I20260928 09:08:01.579005 17294 raft_consensus.cc:2726] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:08:02.048296 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:02.048149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:46603)
0928 09:08:02.048219 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:02.048256 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 09:08:02.109365 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:02.109098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:40167)
0928 09:08:02.109271 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:02.109315 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260928 09:08:02.238767 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:02.238548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:33625)
0928 09:08:02.238692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:02.238729 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260928 09:08:03.048996 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:03.048772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:50861)
0928 09:08:03.048895 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:03.048934 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 09:08:03.110023 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:03.109818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:43589)
0928 09:08:03.109935 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:03.109970 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260928 09:08:03.239500 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:03.239287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:38869)
0928 09:08:03.239413 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:03.239455 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260928 09:08:03.437213 17304 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:03.437306 17304 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:08:03.437512 17304 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
W20260928 09:08:03.438014 16907 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
W20260928 09:08:03.438081 16910 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:08:03.438123 16910 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb; no voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b
I20260928 09:08:03.438221 17304 raft_consensus.cc:2726] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:08:04.049791 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:04.049511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:49291)
0928 09:08:04.049702 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:04.049743 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260928 09:08:04.110824 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:04.110579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:45317)
0928 09:08:04.110744 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:04.110781 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260928 09:08:04.240266 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:04.240057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:60543)
0928 09:08:04.240181 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:04.240225 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 09:08:05.050534 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:05.050310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:58141)
0928 09:08:05.050443 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:05.050483 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260928 09:08:05.111629 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:05.111369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:56321)
0928 09:08:05.111528 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:05.111571 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260928 09:08:05.241083 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:05.240847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:46575)
0928 09:08:05.241002 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:05.241041 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260928 09:08:06.051432 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:06.051191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:59589)
0928 09:08:06.051339 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:06.051383 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 09:08:06.112452 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:06.112185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:57507)
0928 09:08:06.112354 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:06.112395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
I20260928 09:08:06.241874 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:06.241643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:34827)
0928 09:08:06.241788 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:06.241829 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260928 09:08:06.580668 17314 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:06.580782 17314 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:08:06.580983 17314 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
W20260928 09:08:06.581346 16910 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
W20260928 09:08:06.581436 16907 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:08:06.581465 16910 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb; no voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b
I20260928 09:08:06.581574 17314 raft_consensus.cc:2726] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:08:07.052177 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:07.051923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:47991)
0928 09:08:07.052076 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:07.052112 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260928 09:08:07.113202 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:07.112964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:51619)
0928 09:08:07.113111 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:07.113155 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260928 09:08:07.242571 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:07.242356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:41393)
0928 09:08:07.242491 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:07.242527 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260928 09:08:08.052881 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:08.052672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:34283)
0928 09:08:08.052794 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:08.052832 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 09:08:08.113984 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:08.113770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:46855)
0928 09:08:08.113893 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:08.113935 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 09:08:08.243287 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:08.243041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:52853)
0928 09:08:08.243177 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:08.243243 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260928 09:08:09.053763 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:09.053433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:39327)
0928 09:08:09.053649 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:09.053686 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260928 09:08:09.114655 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:09.114440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:36097)
0928 09:08:09.114563 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:09.114607 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 09:08:09.118285 17323 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:09.118389 17323 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:08:09.118582 17323 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
W20260928 09:08:09.118942 16907 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
W20260928 09:08:09.119004 16910 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:08:09.119035 16910 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb; no voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b
I20260928 09:08:09.119104 17323 raft_consensus.cc:2726] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:08:09.243975 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:09.243718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:60043)
0928 09:08:09.243871 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:09.243912 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260928 09:08:10.054558 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:10.054285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:44199)
0928 09:08:10.054465 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:10.054508 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260928 09:08:10.115334 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:10.115094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:57267)
0928 09:08:10.115243 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:10.115286 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 09:08:10.244737 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:10.244498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:49965)
0928 09:08:10.244623 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:10.244689 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260928 09:08:11.055299 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:11.055044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:42951)
0928 09:08:11.055207 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:11.055249 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260928 09:08:11.103531 17329 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:11.103627 17329 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:08:11.103902 17329 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
W20260928 09:08:11.104319 16907 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
W20260928 09:08:11.104399 16910 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:08:11.104434 16910 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb; no voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b
I20260928 09:08:11.104512 17329 raft_consensus.cc:2726] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:08:11.116205 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:11.115937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:52311)
0928 09:08:11.116103 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:11.116152 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260928 09:08:11.245433 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:11.245206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:57919)
0928 09:08:11.245348 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:11.245387 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260928 09:08:12.056070 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:12.055807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:54471)
0928 09:08:12.055979 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:12.056022 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260928 09:08:12.116928 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:12.116698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:46857)
0928 09:08:12.116835 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:12.116881 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260928 09:08:12.246130 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:12.245908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:56573)
0928 09:08:12.246043 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:12.246083 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260928 09:08:13.056891 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:13.056610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:60387)
0928 09:08:13.056752 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:13.056793 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260928 09:08:13.117787 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:13.117498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:39031)
0928 09:08:13.117690 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:13.117738 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":126,"threads_started":1}
I20260928 09:08:13.246762 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:13.246560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:48265)
0928 09:08:13.246678 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:13.246713 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260928 09:08:14.057683 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:14.057443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:37519)
0928 09:08:14.057601 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:14.057634 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260928 09:08:14.118485 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:14.118258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:42509)
0928 09:08:14.118397 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:14.118439 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260928 09:08:14.247514 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:14.247277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:37587)
0928 09:08:14.247425 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:14.247462 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 09:08:14.896919 17341 raft_consensus.cc:482] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:14.897011 17341 raft_consensus.cc:504] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797f41f873a34a978f2dfa0e8f29ef0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39183 } } peers { permanent_uuid: "30baa5450fe84daeb490c0e31f2f7eb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36065 } } peers { permanent_uuid: "4b5e5d0eb258464896d8da3789dc3adb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } }
I20260928 09:08:14.897235 17341 leader_election.cc:288] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183), 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065)
W20260928 09:08:14.897671 16907 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30baa5450fe84daeb490c0e31f2f7eb8 (127.0.98.3:36065): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
W20260928 09:08:14.897789 16910 leader_election.cc:341] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 797f41f873a34a978f2dfa0e8f29ef0b (127.0.98.2:39183): Not found: Tablet not found: f484ff5e51a5479784ea2a7931ff4f40
I20260928 09:08:14.897835 16910 leader_election.cc:302] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e5d0eb258464896d8da3789dc3adb; no voters: 30baa5450fe84daeb490c0e31f2f7eb8, 797f41f873a34a978f2dfa0e8f29ef0b
I20260928 09:08:14.897914 17341 raft_consensus.cc:2726] T f484ff5e51a5479784ea2a7931ff4f40 P 4b5e5d0eb258464896d8da3789dc3adb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:08:15.058492 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:15.058209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.1:53467)
0928 09:08:15.058392 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:15.058435 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260928 09:08:15.119334 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:15.119055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.2:43141)
0928 09:08:15.119229 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:15.119270 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260928 09:08:15.241385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 16858
I20260928 09:08:15.246301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17025
I20260928 09:08:15.248216 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:15.247992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43785 (local address 127.0.98.3:59243)
0928 09:08:15.248113 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:15.248155 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260928 09:08:15.250912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17158
2026-09-28T09:08:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58472 ms)
[----------] 30 tests from SecurityITest (233692 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:15 dist-test-slave-smns krb5kdc[17347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:15 dist-test-slave-smns krb5kdc[17347](info): set up 2 sockets
Sep 28 09:08:15 dist-test-slave-smns krb5kdc[17347](info): commencing operation
krb5kdc: starting...
W20260928 09:08:16.301254 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:16.420965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:16.502560 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:16.502801 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:16.502840 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:16.504000 17360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:16.504069 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:16.504101 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:16.504130 17360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:16.504160 17360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:16.505918 17360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:16.506212 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:16.506439 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:16.512337 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:16.512390 17360 server_base.cc:1034] running on GCE node
W20260928 09:08:16.512358 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:16.512373 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:16.512844 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:16.513135 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:16.514295 17360 hybrid_clock.cc:648] HybridClock initialized: now 1790586496514275 us; error 39 us; skew 500 ppm
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:16.516872 17360 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:16.517352 17360 webserver.cc:460] Webserver started at http://127.0.98.62:42591/ using document root <none> and password file <none>
I20260928 09:08:16.517627 17360 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:16.517696 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:16.517845 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:16.518966 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "f8188728de76479b960b4afab0bfcfb5"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "a8b8a45354ce65d81b53fd0764a02cd4"
server_key_iv: "9533719a1a459304167a9f39fdd668c4"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.519335 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8188728de76479b960b4afab0bfcfb5"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "a8b8a45354ce65d81b53fd0764a02cd4"
server_key_iv: "9533719a1a459304167a9f39fdd668c4"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.520656 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:16.521348 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.521516 17360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:16.521632 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "f8188728de76479b960b4afab0bfcfb5"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "a8b8a45354ce65d81b53fd0764a02cd4"
server_key_iv: "9533719a1a459304167a9f39fdd668c4"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.521721 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:16.533677 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:16.535432 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:16.535583 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:16.539449 17360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42771
I20260928 09:08:16.539490 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42771 every 8 connection(s)
I20260928 09:08:16.539834 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:16.540678 17428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:16.542984 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5: Bootstrap starting.
I20260928 09:08:16.543505 17428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:16.543747 17428 log.cc:824] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:16.544378 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5: No bootstrap required, opened a new log
I20260928 09:08:16.545506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17360
I20260928 09:08:16.545640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:16.545861 392 external_mini_cluster.cc:1428] Setting key 82928e797ee44ff23179d72d4e8a06fe
I20260928 09:08:16.546227 17428 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8188728de76479b960b4afab0bfcfb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42771 } }
I20260928 09:08:16.546455 17428 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:16.546514 17428 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8188728de76479b960b4afab0bfcfb5, State: Initialized, Role: FOLLOWER
I20260928 09:08:16.546666 17428 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8188728de76479b960b4afab0bfcfb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42771 } }
I20260928 09:08:16.546765 17428 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:16.546795 17428 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:16.546842 17428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:16.547855 17428 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8188728de76479b960b4afab0bfcfb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42771 } }
I20260928 09:08:16.547986 17428 leader_election.cc:302] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8188728de76479b960b4afab0bfcfb5; no voters:
I20260928 09:08:16.548143 17428 leader_election.cc:288] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:16.548197 17433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:16.548338 17433 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [term 1 LEADER]: Becoming Leader. State: Replica: f8188728de76479b960b4afab0bfcfb5, State: Running, Role: LEADER
I20260928 09:08:16.548391 17428 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:16.548460 17433 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8188728de76479b960b4afab0bfcfb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42771 } }
I20260928 09:08:16.548933 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8188728de76479b960b4afab0bfcfb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8188728de76479b960b4afab0bfcfb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42771 } } }
I20260928 09:08:16.548925 17433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8188728de76479b960b4afab0bfcfb5. Latest consensus state: current_term: 1 leader_uuid: "f8188728de76479b960b4afab0bfcfb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8188728de76479b960b4afab0bfcfb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42771 } } }
I20260928 09:08:16.549037 17433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:16.549031 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:16.549667 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:16.550655 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:16.552467 17441 catalog_manager.cc:1347] Generated new cluster ID: 8cf5680abd854f00934f4fe8efb3cc53
I20260928 09:08:16.552528 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:16.558198 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:16.546426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.0.98.62:42771)
0928 09:08:16.546649 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:16.546658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:16.546692 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:08:16.546811 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:16.546819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:16.546877 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:16.547029 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:16.547709 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.549459 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.550636 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.551219 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.551671 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:16.554154 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:16.554159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:16.554159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:16.554172 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:16.556147 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:16.556527 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:16.556530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:16.556531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:16.556662 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:16.556854 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:16.556855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:16.556855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:16.557163 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:16.557771 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:16.557845 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 09:08:16.557995 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260928 09:08:16.584779 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:16.585624 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:16.589586 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5: Generated new TSK 0
I20260928 09:08:16.589776 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:08:16.620940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42771
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:16.703817 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:16.704025 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:16.704046 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:16.705309 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:16.705353 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:16.705380 17454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:08:16.706935 17454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:16.707197 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:16.707441 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:16.712683 17462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:16.712678 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:16.712713 17454 server_base.cc:1034] running on GCE node
W20260928 09:08:16.712682 17460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:16.713196 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:16.713400 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:16.714540 17454 hybrid_clock.cc:648] HybridClock initialized: now 1790586496714524 us; error 34 us; skew 500 ppm
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:16.717049 17454 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:08:16.717458 17454 webserver.cc:460] Webserver started at http://127.0.98.1:40225/ using document root <none> and password file <none>
I20260928 09:08:16.717720 17454 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:16.717767 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:16.717855 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:16.718837 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d298ca3daf546c1ac079d64245f1696"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "e6d1d7a4040ab3798f54feb5093f91af"
server_key_iv: "557cfc7f5ea6b04bf0cb7e34c583658c"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.719120 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d298ca3daf546c1ac079d64245f1696"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "e6d1d7a4040ab3798f54feb5093f91af"
server_key_iv: "557cfc7f5ea6b04bf0cb7e34c583658c"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.720350 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:16.721031 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.721199 17454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:16.721298 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "4d298ca3daf546c1ac079d64245f1696"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "e6d1d7a4040ab3798f54feb5093f91af"
server_key_iv: "557cfc7f5ea6b04bf0cb7e34c583658c"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.721369 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:16.731865 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:16.733618 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:16.733811 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:16.734036 17454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:16.734329 17454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:16.734378 17454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.734418 17454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:16.734447 17454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.740011 17454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46341
I20260928 09:08:16.740080 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46341 every 8 connection(s)
I20260928 09:08:16.740387 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:08:16.745828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17454
I20260928 09:08:16.746078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:08:16.746326 392 external_mini_cluster.cc:1428] Setting key ccfbfd8e2e209953a57ed49f2315bb85
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:16.750574 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:16.741098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60285 (local address 127.0.98.62:42771)
0928 09:08:16.741223 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:16.741225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:16.741348 (+ 123us) server_negotiation.cc:407] Connection header received
0928 09:08:16.742353 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:16.742355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:16.742364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:16.742423 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:16.743045 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.743629 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.744657 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.744867 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.744928 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:16.747375 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:16.747376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:16.747376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:16.747383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:16.748976 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:16.749444 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:16.749446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:16.749446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:16.749509 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:16.749827 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:16.749828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:16.749828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:16.750004 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:16.750076 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:16.750389 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 09:08:16.750479 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 09:08:16.750813 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:16.741058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42771 (local address 127.0.98.1:60285)
0928 09:08:16.741225 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:16.741268 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:16.742256 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:16.742480 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:16.742484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:16.742745 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:16.742936 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:16.742940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.743719 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.743720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:16.744560 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:16.744563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.744945 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.744945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:16.745049 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:16.745589 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:16.745599 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:16.747289 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:16.749055 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:16.749062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:16.749064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:16.749355 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:16.749602 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:16.749603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:16.749603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:16.749733 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:16.750072 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:16.750077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:16.750198 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 09:08:16.750348 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 09:08:16.750538 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260928 09:08:16.751305 17583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42771
I20260928 09:08:16.751400 17583 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:16.751595 17583 heartbeater.cc:511] Master 127.0.98.62:42771 requested a full tablet report, sending...
I20260928 09:08:16.752192 17392 ts_manager.cc:194] Registered new tserver with Master: 4d298ca3daf546c1ac079d64245f1696 (127.0.98.1:46341)
I20260928 09:08:16.753203 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60285
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:08:16.771893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42771
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:16.856585 17588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:16.856786 17588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:16.856808 17588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:16.857969 17588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:16.858011 17588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:16.858038 17588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:16.859479 17588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:16.859668 17588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:16.859838 17588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:16.864785 17596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:16.864785 17594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:16.864832 17593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:16.864879 17588 server_base.cc:1034] running on GCE node
I20260928 09:08:16.865281 17588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:16.865480 17588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:16.866649 17588 hybrid_clock.cc:648] HybridClock initialized: now 1790586496866636 us; error 31 us; skew 500 ppm
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:16.869242 17588 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:08:16.869740 17588 webserver.cc:460] Webserver started at http://127.0.98.2:44055/ using document root <none> and password file <none>
I20260928 09:08:16.869927 17588 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:16.869969 17588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:16.870065 17588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:16.871048 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c3edaa548fd4118a3fff5fa269c0a1b"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "356f240e90af16e05772252954832a40"
server_key_iv: "fb56dbd5ba8c7d45d5c35187a31621ad"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.871352 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c3edaa548fd4118a3fff5fa269c0a1b"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "356f240e90af16e05772252954832a40"
server_key_iv: "fb56dbd5ba8c7d45d5c35187a31621ad"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.872597 17588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:16.873283 17603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.873466 17588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:16.873586 17588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "8c3edaa548fd4118a3fff5fa269c0a1b"
format_stamp: "Formatted at 2026-09-28 09:08:16 on dist-test-slave-smns"
server_key: "356f240e90af16e05772252954832a40"
server_key_iv: "fb56dbd5ba8c7d45d5c35187a31621ad"
server_key_version: "encryptionkey@0"
I20260928 09:08:16.873657 17588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:16.883447 17588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:16.885143 17588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:16.885291 17588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:16.885490 17588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:16.885818 17588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:16.885857 17588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.885898 17588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:16.885926 17588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:16.891695 17588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41867
I20260928 09:08:16.891757 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41867 every 8 connection(s)
I20260928 09:08:16.892033 17588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:08:16.896726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17588
I20260928 09:08:16.896814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:08:16.897027 392 external_mini_cluster.cc:1428] Setting key 1f450e24ba853cca7d580f037ea9006a
Sep 28 09:08:16 dist-test-slave-smns krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:16.902447 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:16.892837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44825 (local address 127.0.98.62:42771)
0928 09:08:16.892897 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:16.892899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:16.893097 (+ 198us) server_negotiation.cc:407] Connection header received
0928 09:08:16.894132 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:16.894134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:16.894148 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:16.894208 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:16.894840 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.895340 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.896340 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.896572 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.896646 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:16.899354 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:16.899355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:16.899356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:16.899365 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:16.900784 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:16.901282 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:16.901282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:16.901283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:16.901333 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:16.901686 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:16.901687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:16.901687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:16.901797 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:16.901853 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:16.902222 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 09:08:16.902316 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 09:08:16.903062 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:16.892829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42771 (local address 127.0.98.2:44825)
0928 09:08:16.892986 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:16.893019 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:16.894023 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:16.894293 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:16.894297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:16.894548 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:16.894726 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:16.894731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.895430 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.895431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:16.896245 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:16.896247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:16.896684 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:16.896686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:16.896823 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:16.897382 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:16.897390 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:16.899240 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:16.900867 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:16.900874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:16.900876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:16.901202 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:16.901399 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:16.901399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:16.901400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:16.901606 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:16.901866 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:16.901874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:16.902043 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 09:08:16.902561 (+ 518us) client_negotiation.cc:240] Negotiation successful
0928 09:08:16.902776 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 09:08:16.903610 17717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42771
I20260928 09:08:16.903707 17717 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:16.903903 17717 heartbeater.cc:511] Master 127.0.98.62:42771 requested a full tablet report, sending...
I20260928 09:08:16.904263 17392 ts_manager.cc:194] Registered new tserver with Master: 8c3edaa548fd4118a3fff5fa269c0a1b (127.0.98.2:41867)
I20260928 09:08:16.904601 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44825
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:08:16.924619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42771
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:17.006232 17722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:17.006438 17722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:17.006460 17722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:17.007591 17722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:17.007630 17722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:17.007656 17722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:17.009219 17722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:17.009426 17722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:17.009626 17722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:17.014353 17730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.014348 17727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.014379 17728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.014719 17722 server_base.cc:1034] running on GCE node
I20260928 09:08:17.014941 17722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:17.015254 17722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:17.016423 17722 hybrid_clock.cc:648] HybridClock initialized: now 1790586497016406 us; error 27 us; skew 500 ppm
Sep 28 09:08:17 dist-test-slave-smns krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:17.019209 17722 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:08:17.019660 17722 webserver.cc:460] Webserver started at http://127.0.98.3:40567/ using document root <none> and password file <none>
I20260928 09:08:17.019913 17722 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:17.019980 17722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:17.020133 17722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:17.021169 17722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "2aecb80e5436441487bd794c1a823fd9"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "9b778170673b33246cb6de7f6ce096ec"
server_key_iv: "19e68fccf3942246477c57ca189be809"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.021528 17722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2aecb80e5436441487bd794c1a823fd9"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "9b778170673b33246cb6de7f6ce096ec"
server_key_iv: "19e68fccf3942246477c57ca189be809"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.022903 17722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:17.023770 17737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.023957 17722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:08:17.024029 17722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "2aecb80e5436441487bd794c1a823fd9"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "9b778170673b33246cb6de7f6ce096ec"
server_key_iv: "19e68fccf3942246477c57ca189be809"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.024083 17722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:17.035882 17722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:17.038125 17722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:17.038239 17722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:17.038405 17722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:17.038642 17722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:17.038672 17722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.038693 17722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:17.038707 17722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.044114 17722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44895
I20260928 09:08:17.044201 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44895 every 8 connection(s)
I20260928 09:08:17.044508 17722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:08:17.049515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17722
I20260928 09:08:17.049664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:08:17.049853 392 external_mini_cluster.cc:1428] Setting key b15dab5a4d11190e469cf45546cabcc6
Sep 28 09:08:17 dist-test-slave-smns krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:17.054133 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.045239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55349 (local address 127.0.98.62:42771)
0928 09:08:17.045326 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.045328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.045434 (+ 106us) server_negotiation.cc:407] Connection header received
0928 09:08:17.046407 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.046408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.046417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.046471 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.047046 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.047536 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.048641 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.048969 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.049053 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.051556 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.051557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.051557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:17.051564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.052856 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:17.053217 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:17.053217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:17.053218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:17.053282 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:17.053478 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:17.053479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:17.053479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:17.053632 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.053691 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.054004 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.054069 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 09:08:17.054406 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.045174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42771 (local address 127.0.98.3:55349)
0928 09:08:17.045326 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.045363 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.046313 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.046528 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.046530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.046780 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:17.046967 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.046971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.047612 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.047613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.048481 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.048484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.049055 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.049057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.049220 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.049846 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:17.049856 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:17.051475 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:17.052929 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:17.052934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:17.052936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:17.053146 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:17.053335 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:17.053335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:17.053336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:17.053428 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:17.053760 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:17.053766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:17.053884 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.053982 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.054103 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 09:08:17.054931 17851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42771
I20260928 09:08:17.055016 17851 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:17.055157 17851 heartbeater.cc:511] Master 127.0.98.62:42771 requested a full tablet report, sending...
I20260928 09:08:17.055475 17392 ts_manager.cc:194] Registered new tserver with Master: 2aecb80e5436441487bd794c1a823fd9 (127.0.98.3:44895)
I20260928 09:08:17.055861 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55349
I20260928 09:08:17.061774 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:08:17.068655 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.063045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.0.98.62:42771)
0928 09:08:17.063116 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.063118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.063189 (+ 71us) server_negotiation.cc:407] Connection header received
0928 09:08:17.063243 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.063244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.063252 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.063297 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.063784 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.064266 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.064995 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.065167 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.065213 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.066516 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.066516 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.066517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:17.066523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.067718 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:17.068023 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:17.068024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:17.068024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:17.068072 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:17.068214 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:17.068214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:17.068214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:17.068356 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.068432 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.068513 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.068569 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:08:17.070272 17392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:08:17.071849 17392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:08:17.078140 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.074059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.98.2:41867)
0928 09:08:17.074203 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.074208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.074444 (+ 236us) server_negotiation.cc:407] Connection header received
0928 09:08:17.074491 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.074494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.074506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.074579 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.074752 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.075570 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.076872 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.077718 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.077799 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.077860 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.078021 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.078060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260928 09:08:17.078197 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.073955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41867 (local address 127.0.0.1:48290)
0928 09:08:17.074233 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.074271 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.074365 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.074597 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.074600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.074602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.074670 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.074672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.075669 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.075670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.076798 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.076801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.077798 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.077799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.077894 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.077907 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.078004 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.078112 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":79,"threads_started":1}
I20260928 09:08:17.078564 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.074016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46341 (local address 127.0.0.1:36998)
0928 09:08:17.074631 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.074641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.074773 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.074965 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.074966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.074968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.075008 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.075009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.076246 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.076247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.077173 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.077175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.078243 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.078244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.078309 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.078313 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.078365 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.078410 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":13,"thread_start_us":50,"threads_started":1}
I20260928 09:08:17.078575 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.074229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.0.98.1:46341)
0928 09:08:17.074396 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.074401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.074693 (+ 292us) server_negotiation.cc:407] Connection header received
0928 09:08:17.074874 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.074877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.074891 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.074967 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.075056 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.076127 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.077271 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.078073 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.078148 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.078207 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.078458 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.078502 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260928 09:08:17.078637 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.074217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.0.98.3:44895)
0928 09:08:17.074413 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.074418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.074450 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:08:17.074533 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.074535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.074551 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.074741 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.074967 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.076158 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.077448 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.078129 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.078187 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.078247 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.078513 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.078551 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260928 09:08:17.078661 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.074016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44895 (local address 127.0.0.1:39210)
0928 09:08:17.074386 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.074398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.074465 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.074836 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.074836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.074839 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.074885 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.074887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.076263 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.076264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.077377 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.077380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.078277 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.078277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.078330 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.078334 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.078380 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.078410 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":66,"threads_started":1}
I20260928 09:08:17.079388 17785 tablet_service.cc:1467] Processing CreateTablet for tablet 859f0b4e7dea462b8ea79b4d2bdedde2 (DEFAULT_TABLE table=test-table [id=19e216a6a60f4bdd831b61650206dd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:17.079365 17517 tablet_service.cc:1467] Processing CreateTablet for tablet 859f0b4e7dea462b8ea79b4d2bdedde2 (DEFAULT_TABLE table=test-table [id=19e216a6a60f4bdd831b61650206dd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:17.079725 17785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859f0b4e7dea462b8ea79b4d2bdedde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.079737 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859f0b4e7dea462b8ea79b4d2bdedde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.080054 17651 tablet_service.cc:1467] Processing CreateTablet for tablet 859f0b4e7dea462b8ea79b4d2bdedde2 (DEFAULT_TABLE table=test-table [id=19e216a6a60f4bdd831b61650206dd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:17.080355 17651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859f0b4e7dea462b8ea79b4d2bdedde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.082823 17865 tablet_bootstrap.cc:492] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Bootstrap starting.
I20260928 09:08:17.083222 17867 tablet_bootstrap.cc:492] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Bootstrap starting.
I20260928 09:08:17.083542 17865 tablet_bootstrap.cc:654] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.083820 17867 tablet_bootstrap.cc:654] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.084087 17865 log.cc:824] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.084398 17867 log.cc:824] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.084753 17866 tablet_bootstrap.cc:492] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: Bootstrap starting.
I20260928 09:08:17.084832 17865 tablet_bootstrap.cc:492] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: No bootstrap required, opened a new log
I20260928 09:08:17.084909 17865 ts_tablet_manager.cc:1397] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:08:17.085170 17867 tablet_bootstrap.cc:492] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: No bootstrap required, opened a new log
I20260928 09:08:17.085238 17867 ts_tablet_manager.cc:1397] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:17.085407 17866 tablet_bootstrap.cc:654] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.086004 17866 log.cc:824] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.086643 17865 raft_consensus.cc:348] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.086771 17866 tablet_bootstrap.cc:492] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: No bootstrap required, opened a new log
I20260928 09:08:17.086805 17865 raft_consensus.cc:374] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.086830 17866 ts_tablet_manager.cc:1397] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:08:17.086848 17865 raft_consensus.cc:729] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3edaa548fd4118a3fff5fa269c0a1b, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.086961 17865 consensus_queue.cc:260] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.087261 17865 ts_tablet_manager.cc:1428] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 09:08:17.087275 17717 heartbeater.cc:503] Master 127.0.98.62:42771 was elected leader, sending a full tablet report...
I20260928 09:08:17.087437 17867 raft_consensus.cc:348] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.087756 17867 raft_consensus.cc:374] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.087801 17867 raft_consensus.cc:729] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aecb80e5436441487bd794c1a823fd9, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.087904 17867 consensus_queue.cc:260] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.088160 17867 ts_tablet_manager.cc:1428] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:17.088192 17851 heartbeater.cc:503] Master 127.0.98.62:42771 was elected leader, sending a full tablet report...
I20260928 09:08:17.088907 17866 raft_consensus.cc:348] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.089088 17866 raft_consensus.cc:374] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.089115 17866 raft_consensus.cc:729] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d298ca3daf546c1ac079d64245f1696, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.089213 17866 consensus_queue.cc:260] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.089450 17866 ts_tablet_manager.cc:1428] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:17.089485 17583 heartbeater.cc:503] Master 127.0.98.62:42771 was elected leader, sending a full tablet report...
W20260928 09:08:17.142699 17718 tablet.cc:2369] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:08:17.234900 17873 raft_consensus.cc:482] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:17.235064 17873 raft_consensus.cc:504] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.235458 17873 leader_election.cc:288] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c3edaa548fd4118a3fff5fa269c0a1b (127.0.98.2:41867), 2aecb80e5436441487bd794c1a823fd9 (127.0.98.3:44895)
I20260928 09:08:17.238359 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.235647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44895 (local address 127.0.98.1:52695)
0928 09:08:17.235807 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.235820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.235883 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.236046 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.236047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.236048 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.236124 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.236126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.236781 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.236783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.237519 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.237521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.238091 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.238092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.238176 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.238179 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.238245 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.238285 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 09:08:17.238359 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.235737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52695 (local address 127.0.98.3:44895)
0928 09:08:17.235797 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.235800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.235873 (+ 73us) server_negotiation.cc:407] Connection header received
0928 09:08:17.235966 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.235967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.235976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.236044 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.236241 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.236704 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.237642 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.238012 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.238094 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.238105 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.238253 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.238294 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:08:17.238507 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.235793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34945 (local address 127.0.98.2:41867)
0928 09:08:17.235837 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.235839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.236011 (+ 172us) server_negotiation.cc:407] Connection header received
0928 09:08:17.236091 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.236093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.236106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.236140 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.236305 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.236769 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.237768 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.238131 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.238217 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.238226 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.238377 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.238418 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:08:17.238694 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.235745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41867 (local address 127.0.98.1:34945)
0928 09:08:17.235877 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.235921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.236021 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.236155 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.236155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.236157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.236232 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.236234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.236857 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.236858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.237677 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.237679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.238219 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.238220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.238296 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.238299 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.238370 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.238620 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 09:08:17.238701 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f0b4e7dea462b8ea79b4d2bdedde2" candidate_uuid: "4d298ca3daf546c1ac079d64245f1696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aecb80e5436441487bd794c1a823fd9" is_pre_election: true
I20260928 09:08:17.238902 17805 raft_consensus.cc:2454] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d298ca3daf546c1ac079d64245f1696 in term 0.
I20260928 09:08:17.239084 17671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f0b4e7dea462b8ea79b4d2bdedde2" candidate_uuid: "4d298ca3daf546c1ac079d64245f1696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" is_pre_election: true
I20260928 09:08:17.239250 17671 raft_consensus.cc:2454] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d298ca3daf546c1ac079d64245f1696 in term 0.
I20260928 09:08:17.239243 17472 leader_election.cc:302] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aecb80e5436441487bd794c1a823fd9, 4d298ca3daf546c1ac079d64245f1696; no voters:
I20260928 09:08:17.239382 17873 raft_consensus.cc:2781] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:08:17.239455 17873 raft_consensus.cc:482] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:08:17.239491 17873 raft_consensus.cc:3037] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:17.240268 17873 raft_consensus.cc:504] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.240435 17873 leader_election.cc:288] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [CANDIDATE]: Term 1 election: Requested vote from peers 8c3edaa548fd4118a3fff5fa269c0a1b (127.0.98.2:41867), 2aecb80e5436441487bd794c1a823fd9 (127.0.98.3:44895)
I20260928 09:08:17.240597 17671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f0b4e7dea462b8ea79b4d2bdedde2" candidate_uuid: "4d298ca3daf546c1ac079d64245f1696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b"
I20260928 09:08:17.240629 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f0b4e7dea462b8ea79b4d2bdedde2" candidate_uuid: "4d298ca3daf546c1ac079d64245f1696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aecb80e5436441487bd794c1a823fd9"
I20260928 09:08:17.240696 17671 raft_consensus.cc:3037] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:17.240696 17805 raft_consensus.cc:3037] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 0 FOLLOWER]: Advancing to term 1
W20260928 09:08:17.241103 17584 tablet.cc:2369] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:08:17.241492 17805 raft_consensus.cc:2454] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d298ca3daf546c1ac079d64245f1696 in term 1.
I20260928 09:08:17.241675 17472 leader_election.cc:302] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aecb80e5436441487bd794c1a823fd9, 4d298ca3daf546c1ac079d64245f1696; no voters:
I20260928 09:08:17.241787 17873 raft_consensus.cc:2781] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:17.241807 17671 raft_consensus.cc:2454] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d298ca3daf546c1ac079d64245f1696 in term 1.
I20260928 09:08:17.241956 17873 raft_consensus.cc:686] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [term 1 LEADER]: Becoming Leader. State: Replica: 4d298ca3daf546c1ac079d64245f1696, State: Running, Role: LEADER
I20260928 09:08:17.242062 17873 consensus_queue.cc:237] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } }
I20260928 09:08:17.242909 17391 catalog_manager.cc:5495] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d298ca3daf546c1ac079d64245f1696 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4d298ca3daf546c1ac079d64245f1696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d298ca3daf546c1ac079d64245f1696" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46341 } health_report { overall_health: HEALTHY } } }
I20260928 09:08:17.260820 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.258455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.0.98.1:46341)
0928 09:08:17.258522 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.258524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.258533 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:17.258579 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.258580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.258598 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.258655 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:17.258762 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.259213 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.260022 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.260206 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.260259 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.260420 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:17.260517 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:17.260577 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.260653 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.260671 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:08:17.262738 17805 raft_consensus.cc:1261] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 1 FOLLOWER]: Refusing update from remote peer 4d298ca3daf546c1ac079d64245f1696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:17.262738 17671 raft_consensus.cc:1261] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 1 FOLLOWER]: Refusing update from remote peer 4d298ca3daf546c1ac079d64245f1696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:17.263078 17875 consensus_queue.cc:1035] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3edaa548fd4118a3fff5fa269c0a1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:17.263211 17873 consensus_queue.cc:1035] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 4d298ca3daf546c1ac079d64245f1696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aecb80e5436441487bd794c1a823fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:17.265561 17879 mvcc.cc:205] Tried to move back new op lower bound from 7334242292785053696 to 7334242292704120832. Current Snapshot: MvccSnapshot[applied={T|T < 7334242292785053696}]
I20260928 09:08:17.265607 17880 mvcc.cc:205] Tried to move back new op lower bound from 7334242292785053696 to 7334242292704120832. Current Snapshot: MvccSnapshot[applied={T|T < 7334242292785053696}]
I20260928 09:08:17.268093 17392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:17.268209 17392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:17.268941 17392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8188728de76479b960b4afab0bfcfb5: Sending DeleteTablet for 3 replicas of tablet 859f0b4e7dea462b8ea79b4d2bdedde2
I20260928 09:08:17.269338 17651 tablet_service.cc:1514] Processing DeleteTablet for tablet 859f0b4e7dea462b8ea79b4d2bdedde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48290
I20260928 09:08:17.269361 17517 tablet_service.cc:1514] Processing DeleteTablet for tablet 859f0b4e7dea462b8ea79b4d2bdedde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36998
I20260928 09:08:17.269580 17785 tablet_service.cc:1514] Processing DeleteTablet for tablet 859f0b4e7dea462b8ea79b4d2bdedde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39210
I20260928 09:08:17.269605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17454
I20260928 09:08:17.270251 17888 tablet_replica.cc:330] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: stopping tablet replica
I20260928 09:08:17.270351 17888 raft_consensus.cc:2229] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:17.270454 17888 raft_consensus.cc:2258] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:17.271077 17890 tablet_replica.cc:330] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: stopping tablet replica
I20260928 09:08:17.271152 17890 raft_consensus.cc:2229] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:17.271270 17890 raft_consensus.cc:2258] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:17.271840 17888 ts_tablet_manager.cc:1905] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:17.271963 17890 ts_tablet_manager.cc:1905] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:17.274106 17890 ts_tablet_manager.cc:1918] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:17.274106 17888 ts_tablet_manager.cc:1918] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:17.274185 17890 log.cc:1196] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/859f0b4e7dea462b8ea79b4d2bdedde2
I20260928 09:08:17.274260 17888 log.cc:1196] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/859f0b4e7dea462b8ea79b4d2bdedde2
I20260928 09:08:17.274492 17890 ts_tablet_manager.cc:1939] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 2aecb80e5436441487bd794c1a823fd9: Deleting consensus metadata
I20260928 09:08:17.274520 17888 ts_tablet_manager.cc:1939] T 859f0b4e7dea462b8ea79b4d2bdedde2 P 8c3edaa548fd4118a3fff5fa269c0a1b: Deleting consensus metadata
I20260928 09:08:17.274943 17379 catalog_manager.cc:4841] TS 8c3edaa548fd4118a3fff5fa269c0a1b (127.0.98.2:41867): tablet 859f0b4e7dea462b8ea79b4d2bdedde2 (table test-table [id=19e216a6a60f4bdd831b61650206dd99]) successfully deleted
W20260928 09:08:17.274993 17376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46341 (error 108)
I20260928 09:08:17.275045 17378 catalog_manager.cc:4841] TS 2aecb80e5436441487bd794c1a823fd9 (127.0.98.3:44895): tablet 859f0b4e7dea462b8ea79b4d2bdedde2 (table test-table [id=19e216a6a60f4bdd831b61650206dd99]) successfully deleted
I20260928 09:08:17.275494 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.275340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46341 (local address 127.0.0.1:37022)
0928 09:08:17.275435 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.275470 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 09:08:17.275537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17588
W20260928 09:08:17.275638 17376 catalog_manager.cc:4584] TS 4d298ca3daf546c1ac079d64245f1696 (127.0.98.1:46341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 859f0b4e7dea462b8ea79b4d2bdedde2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46341: connect: Connection refused (error 111)
I20260928 09:08:17.280788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17722
I20260928 09:08:17.285431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17360
2026-09-28T09:08:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2033 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T09:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:17Z Disabled control of system clock
I20260928 09:08:17.313177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:17.394550 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:17.394766 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:17.394786 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:17.395862 17895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:17.395900 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:17.395913 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:17.395925 17895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:17.395938 17895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:17.397289 17895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:17.397482 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:17.397799 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:17.402470 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.402486 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.402490 17895 server_base.cc:1034] running on GCE node
W20260928 09:08:17.402474 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.403035 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:17.403375 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:17.404525 17895 hybrid_clock.cc:648] HybridClock initialized: now 1790586497404496 us; error 48 us; skew 500 ppm
I20260928 09:08:17.405843 17895 webserver.cc:460] Webserver started at http://127.0.98.62:38293/ using document root <none> and password file <none>
I20260928 09:08:17.406086 17895 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:17.406132 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:17.406278 17895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:17.407378 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "f61fbcb221154d018e1547fea72a6cf3"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "2df321d18e26514608bc805092d46c1f"
server_key_iv: "667210529c36bbdb5a44254200c5d31b"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.407745 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "f61fbcb221154d018e1547fea72a6cf3"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "2df321d18e26514608bc805092d46c1f"
server_key_iv: "667210529c36bbdb5a44254200c5d31b"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.409085 17895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:17.409845 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.410017 17895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:17.410112 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "f61fbcb221154d018e1547fea72a6cf3"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "2df321d18e26514608bc805092d46c1f"
server_key_iv: "667210529c36bbdb5a44254200c5d31b"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.410193 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:17.437029 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:17.437490 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:17.437705 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:17.441956 17895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39695
I20260928 09:08:17.441988 17961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39695 every 8 connection(s)
I20260928 09:08:17.442348 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:17.443503 17962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.445950 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3: Bootstrap starting.
I20260928 09:08:17.446524 17962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.446821 17962 log.cc:824] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.447557 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3: No bootstrap required, opened a new log
I20260928 09:08:17.447559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17895
I20260928 09:08:17.447702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:17.447911 392 external_mini_cluster.cc:1428] Setting key 07d90bfba40c7b6c2296aa7ab8fe4635
I20260928 09:08:17.449878 17962 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61fbcb221154d018e1547fea72a6cf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260928 09:08:17.450057 17962 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.450093 17962 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61fbcb221154d018e1547fea72a6cf3, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.450254 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61fbcb221154d018e1547fea72a6cf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260928 09:08:17.450320 17962 raft_consensus.cc:388] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:17.450340 17962 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:17.450434 17962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:17.451165 17962 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61fbcb221154d018e1547fea72a6cf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260928 09:08:17.451289 17962 leader_election.cc:302] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61fbcb221154d018e1547fea72a6cf3; no voters:
I20260928 09:08:17.451448 17962 leader_election.cc:288] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:17.451507 17967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:17.451689 17962 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:17.451686 17967 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [term 1 LEADER]: Becoming Leader. State: Replica: f61fbcb221154d018e1547fea72a6cf3, State: Running, Role: LEADER
I20260928 09:08:17.451892 17967 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61fbcb221154d018e1547fea72a6cf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260928 09:08:17.452281 17968 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f61fbcb221154d018e1547fea72a6cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61fbcb221154d018e1547fea72a6cf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } }
I20260928 09:08:17.452363 17968 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:17.452531 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61fbcb221154d018e1547fea72a6cf3. Latest consensus state: current_term: 1 leader_uuid: "f61fbcb221154d018e1547fea72a6cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61fbcb221154d018e1547fea72a6cf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } }
I20260928 09:08:17.452567 17976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:17.452742 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:17.453286 17976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:17.455303 17976 catalog_manager.cc:1347] Generated new cluster ID: 88d2146663804abaad15fe557922ceab
I20260928 09:08:17.455446 17976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:17.458375 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.448442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.98.62:39695)
0928 09:08:17.448689 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.448699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.448742 (+ 43us) server_negotiation.cc:407] Connection header received
0928 09:08:17.449838 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.449847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.449929 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.450082 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.450420 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.452219 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.453528 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.454249 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.454374 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.457643 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.457647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.457647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:17.457847 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.457889 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:17.457902 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.457979 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.458083 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.458166 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20260928 09:08:17.480590 17976 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:17.481361 17976 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:17.486284 17976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3: Generated new TSK 0
I20260928 09:08:17.486493 17976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:17.490191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--builtin_ntp_servers=127.0.98.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:17.575296 17986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:17.575526 17986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:17.575563 17986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:17.576695 17986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:17.576759 17986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:17.576802 17986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:08:17.578208 17986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:17.578466 17986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:17.578759 17986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:17.584540 17994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.584586 17992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.584542 17991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.584900 17986 server_base.cc:1034] running on GCE node
I20260928 09:08:17.585098 17986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:17.585381 17986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:17.586551 17986 hybrid_clock.cc:648] HybridClock initialized: now 1790586497586541 us; error 29 us; skew 500 ppm
I20260928 09:08:17.587834 17986 webserver.cc:460] Webserver started at http://127.0.98.1:38909/ using document root <none> and password file <none>
I20260928 09:08:17.588085 17986 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:17.588132 17986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:17.588299 17986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:17.589392 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "6563e7f630d14bb4a6ff4489577a044c"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "d99c314b3b8307afd4b40b0099f669b3"
server_key_iv: "a0ec1cc4773ca432b1b55ea4461208c5"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.589797 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6563e7f630d14bb4a6ff4489577a044c"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "d99c314b3b8307afd4b40b0099f669b3"
server_key_iv: "a0ec1cc4773ca432b1b55ea4461208c5"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.591176 17986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:17.592000 18000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.592190 17986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:17.592296 17986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "6563e7f630d14bb4a6ff4489577a044c"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "d99c314b3b8307afd4b40b0099f669b3"
server_key_iv: "a0ec1cc4773ca432b1b55ea4461208c5"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.592384 17986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:17.599877 17986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:17.600145 17986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:17.600291 17986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:17.600497 17986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:17.600806 17986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:17.600859 17986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.600900 17986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:17.600929 17986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.607779 17986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39769
I20260928 09:08:17.607820 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39769 every 8 connection(s)
I20260928 09:08:17.608134 17986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:08:17.613317 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.608828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55077 (local address 127.0.98.62:39695)
0928 09:08:17.608888 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.608889 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.609084 (+ 195us) server_negotiation.cc:407] Connection header received
0928 09:08:17.609666 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.609667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.609679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.609737 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.610428 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.610908 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.611890 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.612070 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.612149 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.613099 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.613100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.613100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:17.613104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.613115 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:17.613118 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.613159 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.613231 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.613260 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 09:08:17.613521 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.608819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.98.1:55077)
0928 09:08:17.608983 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.609016 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.609515 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.609794 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.609797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.609805 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:17.610353 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.610359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.610983 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.610983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.611812 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.611815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.612204 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.612205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.612313 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.612957 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:17.612958 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:17.612996 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:17.613029 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:17.613164 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:17.613167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:17.613169 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.613215 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.613290 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 09:08:17.614148 18114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39695
I20260928 09:08:17.614259 18114 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:17.614439 18114 heartbeater.cc:511] Master 127.0.98.62:39695 requested a full tablet report, sending...
I20260928 09:08:17.614959 17926 ts_manager.cc:194] Registered new tserver with Master: 6563e7f630d14bb4a6ff4489577a044c (127.0.98.1:39769)
I20260928 09:08:17.615345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 17986
I20260928 09:08:17.615423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:08:17.615610 392 external_mini_cluster.cc:1428] Setting key f3b61b6111a92d85fe9e212ab3dc4399
I20260928 09:08:17.615698 17926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55077
I20260928 09:08:17.616618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--builtin_ntp_servers=127.0.98.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:17.699777 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:17.699987 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:17.700009 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:17.701243 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:17.701282 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:17.701308 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:17.703294 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:17.703538 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:17.703894 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:08:17.708986 18117 server_base.cc:1034] running on GCE node
W20260928 09:08:17.708933 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.708945 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.708933 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.709417 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:17.709700 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:17.710843 18117 hybrid_clock.cc:648] HybridClock initialized: now 1790586497710831 us; error 30 us; skew 500 ppm
I20260928 09:08:17.712296 18117 webserver.cc:460] Webserver started at http://127.0.98.2:43531/ using document root <none> and password file <none>
I20260928 09:08:17.712536 18117 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:17.712582 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:17.712755 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:17.714025 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a6db0ff40c54eea8c18ed65db6f7905"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "cb1f67a38a16b1331dcac750d80e7d80"
server_key_iv: "baa152dbc198f5c8542a94e097a84d0f"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.714383 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a6db0ff40c54eea8c18ed65db6f7905"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "cb1f67a38a16b1331dcac750d80e7d80"
server_key_iv: "baa152dbc198f5c8542a94e097a84d0f"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.715641 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:17.716365 18131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.716524 18117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:17.716619 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "3a6db0ff40c54eea8c18ed65db6f7905"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "cb1f67a38a16b1331dcac750d80e7d80"
server_key_iv: "baa152dbc198f5c8542a94e097a84d0f"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.716706 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:17.742741 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:17.743072 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:17.743237 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:17.743503 18117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:17.743857 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:17.743912 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.743954 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:17.743983 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.749555 18117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39343
I20260928 09:08:17.749639 18244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39343 every 8 connection(s)
I20260928 09:08:17.750008 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:08:17.751683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18117
I20260928 09:08:17.751788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:08:17.752017 392 external_mini_cluster.cc:1428] Setting key e1354d89a03c9b1937e0ed7af22457aa
I20260928 09:08:17.752986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--builtin_ntp_servers=127.0.98.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 09:08:17.755883 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.750795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42721 (local address 127.0.98.62:39695)
0928 09:08:17.750866 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.750868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.751130 (+ 262us) server_negotiation.cc:407] Connection header received
0928 09:08:17.751670 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.751671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.751685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.751773 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.752593 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.753180 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.754227 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.754552 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.754595 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.755617 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.755619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.755619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:17.755623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.755636 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:17.755640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.755689 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.755778 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.755806 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:08:17.756103 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.750793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.98.2:42721)
0928 09:08:17.750977 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.751019 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.751520 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.751857 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.751861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.751872 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:17.752497 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.752503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.753258 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.753259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.754111 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.754114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.754699 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.754700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.754864 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.755452 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:17.755452 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:17.755480 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:17.755524 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:17.755688 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:17.755694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:17.755697 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.755780 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.755880 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 09:08:17.756592 18245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39695
I20260928 09:08:17.756690 18245 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:17.756870 18245 heartbeater.cc:511] Master 127.0.98.62:39695 requested a full tablet report, sending...
I20260928 09:08:17.757270 17925 ts_manager.cc:194] Registered new tserver with Master: 3a6db0ff40c54eea8c18ed65db6f7905 (127.0.98.2:39343)
I20260928 09:08:17.757710 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42721
W20260928 09:08:17.835839 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:17.836045 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:17.836066 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:17.837268 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:17.837308 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:17.837333 18248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:17.838786 18248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:17.839028 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:17.839342 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:17.844532 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:17.844552 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.844499 18248 server_base.cc:1034] running on GCE node
W20260928 09:08:17.844508 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:17.845038 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:17.845304 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:17.846475 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790586497846472 us; error 40 us; skew 500 ppm
I20260928 09:08:17.847842 18248 webserver.cc:460] Webserver started at http://127.0.98.3:39587/ using document root <none> and password file <none>
I20260928 09:08:17.848114 18248 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:17.848198 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:17.848356 18248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:17.849535 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "1968567fade84fd0a0cb6b77332addc7"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "a094b0aa6fabc82c8a242e91d7b0290b"
server_key_iv: "47d4a63ee7f1018a9e8cdb266fcbb4c8"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.849977 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1968567fade84fd0a0cb6b77332addc7"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "a094b0aa6fabc82c8a242e91d7b0290b"
server_key_iv: "47d4a63ee7f1018a9e8cdb266fcbb4c8"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.851356 18248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:17.852146 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.852420 18248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:17.852527 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "1968567fade84fd0a0cb6b77332addc7"
format_stamp: "Formatted at 2026-09-28 09:08:17 on dist-test-slave-smns"
server_key: "a094b0aa6fabc82c8a242e91d7b0290b"
server_key_iv: "47d4a63ee7f1018a9e8cdb266fcbb4c8"
server_key_version: "encryptionkey@0"
I20260928 09:08:17.852587 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:17.869325 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:17.869663 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:17.869772 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:17.869967 18248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:17.870260 18248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:17.870291 18248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.870313 18248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:17.870326 18248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:17.875715 18248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33731
I20260928 09:08:17.875803 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33731 every 8 connection(s)
I20260928 09:08:17.876102 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:08:17.878508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18248
I20260928 09:08:17.878618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:08:17.878852 392 external_mini_cluster.cc:1428] Setting key 8abe9a804581e206a00e04bbfd9a0321
I20260928 09:08:17.881842 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.876882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35979 (local address 127.0.98.62:39695)
0928 09:08:17.876955 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.876957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.877180 (+ 223us) server_negotiation.cc:407] Connection header received
0928 09:08:17.877741 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.877742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.877753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.877813 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.878490 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.879223 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.880486 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.880691 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.880758 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.881503 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.881504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.881504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:17.881509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.881523 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:17.881527 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.881661 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.881750 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.881777 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:08:17.881987 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.876807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.98.3:35979)
0928 09:08:17.877006 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.877048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.877597 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.877878 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.877881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.877890 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:17.878384 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.878388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.879352 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.879352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.880408 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.880411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.880775 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.880776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.880885 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.881373 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:17.881373 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:17.881401 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:17.881425 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:17.881665 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:17.881668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:17.881671 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.881722 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.881800 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260928 09:08:17.882490 18376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39695
I20260928 09:08:17.882583 18376 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:17.882786 18376 heartbeater.cc:511] Master 127.0.98.62:39695 requested a full tablet report, sending...
I20260928 09:08:17.883154 17925 ts_manager.cc:194] Registered new tserver with Master: 1968567fade84fd0a0cb6b77332addc7 (127.0.98.3:33731)
I20260928 09:08:17.883549 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35979
I20260928 09:08:17.890697 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:08:17.895390 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.892286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.0.98.62:39695)
0928 09:08:17.892371 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.892373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.892493 (+ 120us) server_negotiation.cc:407] Connection header received
0928 09:08:17.892581 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.892582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.892597 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.892658 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:17.892861 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.893349 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.894172 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.894377 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.894453 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.895108 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:17.895109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:17.895110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:17.895115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:17.895128 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:17.895131 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:17.895215 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.895276 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.895300 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:08:17.897082 17925 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:08:17.898765 17925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:08:17.905561 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.901412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33731 (local address 127.0.0.1:58690)
0928 09:08:17.901626 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.901637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.901683 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.902100 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.902102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.902104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.902143 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.902144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.903057 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.903058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.904244 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.904246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.905240 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.905241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.905322 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.905336 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.905412 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.905462 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":68,"threads_started":1}
I20260928 09:08:17.905593 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.901631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.0.98.3:33731)
0928 09:08:17.901816 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.901820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.901836 (+ 16us) server_negotiation.cc:407] Connection header received
0928 09:08:17.901890 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.901892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.901909 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.901986 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.902236 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.902969 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.904318 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.905144 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.905229 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.905330 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.905427 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.905474 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260928 09:08:17.906112 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.901276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.1:39769)
0928 09:08:17.901430 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.901435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.901601 (+ 166us) server_negotiation.cc:407] Connection header received
0928 09:08:17.901793 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.901796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.901813 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.901901 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.902072 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.903380 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.904553 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.905571 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.905645 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.905789 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.905989 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.906027 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260928 09:08:17.906095 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.901156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39769 (local address 127.0.0.1:43000)
0928 09:08:17.901471 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.901515 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.901633 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.901914 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.901916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.901919 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.901989 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.901991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.903468 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.903469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.904475 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.904476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.905834 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.905835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.905886 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.905890 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.905974 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.906015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":107,"threads_started":1}
I20260928 09:08:17.906564 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.901107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39343 (local address 127.0.0.1:36570)
0928 09:08:17.901468 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:17.901514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:17.901629 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:17.901976 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:17.901977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:17.901979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:17.902059 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.902061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.904544 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.904544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.905232 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:17.905234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.906323 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.906325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:17.906389 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.906393 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:17.906450 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:08:17.906495 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":152,"threads_started":1}
I20260928 09:08:17.906774 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:17.901284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.0.98.2:39343)
0928 09:08:17.901523 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:17.901527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:17.901758 (+ 231us) server_negotiation.cc:407] Connection header received
0928 09:08:17.901805 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:17.901808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:17.901823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:17.901901 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:17.902151 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.903371 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.905324 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:17.906107 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:17.906191 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:17.906290 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:17.906662 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 09:08:17.906701 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260928 09:08:17.906953 18048 tablet_service.cc:1467] Processing CreateTablet for tablet 577be77dd3ca42d281a821a456b5f97a (DEFAULT_TABLE table=test-table [id=d0bfd3b1bfda4800985f1e2e42824dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:17.907264 18048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577be77dd3ca42d281a821a456b5f97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.907145 18310 tablet_service.cc:1467] Processing CreateTablet for tablet 577be77dd3ca42d281a821a456b5f97a (DEFAULT_TABLE table=test-table [id=d0bfd3b1bfda4800985f1e2e42824dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:17.907518 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577be77dd3ca42d281a821a456b5f97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.909833 18390 tablet_bootstrap.cc:492] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: Bootstrap starting.
I20260928 09:08:17.910508 18391 tablet_bootstrap.cc:492] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Bootstrap starting.
I20260928 09:08:17.910545 18390 tablet_bootstrap.cc:654] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.910432 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 577be77dd3ca42d281a821a456b5f97a (DEFAULT_TABLE table=test-table [id=d0bfd3b1bfda4800985f1e2e42824dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:17.910780 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577be77dd3ca42d281a821a456b5f97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:17.910866 18390 log.cc:824] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.911315 18391 tablet_bootstrap.cc:654] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.911612 18391 log.cc:824] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.911682 18390 tablet_bootstrap.cc:492] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: No bootstrap required, opened a new log
I20260928 09:08:17.911757 18390 ts_tablet_manager.cc:1397] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:17.912331 18391 tablet_bootstrap.cc:492] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: No bootstrap required, opened a new log
I20260928 09:08:17.912540 18391 ts_tablet_manager.cc:1397] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:08:17.914170 18394 tablet_bootstrap.cc:492] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Bootstrap starting.
I20260928 09:08:17.914404 18390 raft_consensus.cc:348] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:17.914565 18390 raft_consensus.cc:374] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.914606 18390 raft_consensus.cc:729] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6563e7f630d14bb4a6ff4489577a044c, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.914721 18390 consensus_queue.cc:260] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:17.914981 18394 tablet_bootstrap.cc:654] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:17.915001 18390 ts_tablet_manager.cc:1428] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:08:17.915027 18114 heartbeater.cc:503] Master 127.0.98.62:39695 was elected leader, sending a full tablet report...
I20260928 09:08:17.915315 18394 log.cc:824] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:17.915475 18391 raft_consensus.cc:348] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:17.915673 18391 raft_consensus.cc:374] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.915742 18391 raft_consensus.cc:729] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1968567fade84fd0a0cb6b77332addc7, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.915936 18391 consensus_queue.cc:260] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:17.916193 18394 tablet_bootstrap.cc:492] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: No bootstrap required, opened a new log
I20260928 09:08:17.916265 18394 ts_tablet_manager.cc:1397] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:17.916256 18391 ts_tablet_manager.cc:1428] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 09:08:17.916406 18376 heartbeater.cc:503] Master 127.0.98.62:39695 was elected leader, sending a full tablet report...
I20260928 09:08:17.918359 18394 raft_consensus.cc:348] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:17.918481 18394 raft_consensus.cc:374] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:17.918502 18394 raft_consensus.cc:729] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6db0ff40c54eea8c18ed65db6f7905, State: Initialized, Role: FOLLOWER
I20260928 09:08:17.918608 18394 consensus_queue.cc:260] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:17.918859 18394 ts_tablet_manager.cc:1428] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:17.918923 18245 heartbeater.cc:503] Master 127.0.98.62:39695 was elected leader, sending a full tablet report...
W20260928 09:08:18.000638 18246 tablet.cc:2369] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:08:18.044265 18395 raft_consensus.cc:482] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:18.044395 18395 raft_consensus.cc:504] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:18.044734 18395 leader_election.cc:288] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a6db0ff40c54eea8c18ed65db6f7905 (127.0.98.2:39343), 1968567fade84fd0a0cb6b77332addc7 (127.0.98.3:33731)
I20260928 09:08:18.047721 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:18.045099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58557 (local address 127.0.98.2:39343)
0928 09:08:18.045219 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:18.045221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:18.045230 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:18.045279 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:18.045281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:18.045293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:18.045365 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:18.045575 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.046055 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.046991 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.047369 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.047455 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:18.047465 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:18.047613 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 09:08:18.047647 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:08:18.047721 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:18.044994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39343 (local address 127.0.98.1:58557)
0928 09:08:18.045060 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:18.045074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:18.045124 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:18.045389 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:18.045390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:18.045393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:18.045470 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:18.045472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.046153 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.046154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:18.046894 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:18.046896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.047455 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.047456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:18.047527 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:18.047530 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:18.047606 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:08:18.047645 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 09:08:18.047919 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:18.045302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32873 (local address 127.0.98.3:33731)
0928 09:08:18.045374 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:18.045376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:18.045387 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:08:18.045439 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:18.045440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:18.045458 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:18.045530 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:18.045742 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.046269 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.047180 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.047539 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.047622 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:18.047632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:18.047780 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 09:08:18.047825 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:08:18.048043 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:18.045160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33731 (local address 127.0.98.1:32873)
0928 09:08:18.045306 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:18.045316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:18.045385 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:18.045589 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:18.045590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:18.045592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:18.045659 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:18.045660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.046366 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.046367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:18.047096 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:18.047097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.047633 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.047634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:18.047701 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:18.047704 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:18.047763 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 09:08:18.047945 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 09:08:18.048352 18199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577be77dd3ca42d281a821a456b5f97a" candidate_uuid: "6563e7f630d14bb4a6ff4489577a044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" is_pre_election: true
I20260928 09:08:18.048377 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577be77dd3ca42d281a821a456b5f97a" candidate_uuid: "6563e7f630d14bb4a6ff4489577a044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1968567fade84fd0a0cb6b77332addc7" is_pre_election: true
I20260928 09:08:18.048542 18199 raft_consensus.cc:2454] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6563e7f630d14bb4a6ff4489577a044c in term 0.
I20260928 09:08:18.048554 18330 raft_consensus.cc:2454] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6563e7f630d14bb4a6ff4489577a044c in term 0.
I20260928 09:08:18.048782 18002 leader_election.cc:302] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1968567fade84fd0a0cb6b77332addc7, 6563e7f630d14bb4a6ff4489577a044c; no voters:
I20260928 09:08:18.048907 18395 raft_consensus.cc:2781] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:08:18.048997 18395 raft_consensus.cc:482] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:08:18.049029 18395 raft_consensus.cc:3037] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:18.049773 18395 raft_consensus.cc:504] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:18.049916 18395 leader_election.cc:288] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [CANDIDATE]: Term 1 election: Requested vote from peers 3a6db0ff40c54eea8c18ed65db6f7905 (127.0.98.2:39343), 1968567fade84fd0a0cb6b77332addc7 (127.0.98.3:33731)
I20260928 09:08:18.050103 18199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577be77dd3ca42d281a821a456b5f97a" candidate_uuid: "6563e7f630d14bb4a6ff4489577a044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6db0ff40c54eea8c18ed65db6f7905"
I20260928 09:08:18.050138 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577be77dd3ca42d281a821a456b5f97a" candidate_uuid: "6563e7f630d14bb4a6ff4489577a044c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1968567fade84fd0a0cb6b77332addc7"
I20260928 09:08:18.050190 18199 raft_consensus.cc:3037] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:18.050215 18330 raft_consensus.cc:3037] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:18.051095 18199 raft_consensus.cc:2454] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6563e7f630d14bb4a6ff4489577a044c in term 1.
I20260928 09:08:18.051095 18330 raft_consensus.cc:2454] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6563e7f630d14bb4a6ff4489577a044c in term 1.
I20260928 09:08:18.051291 18002 leader_election.cc:302] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a6db0ff40c54eea8c18ed65db6f7905, 6563e7f630d14bb4a6ff4489577a044c; no voters:
I20260928 09:08:18.051417 18395 raft_consensus.cc:2781] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:18.051599 18395 raft_consensus.cc:686] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 1 LEADER]: Becoming Leader. State: Replica: 6563e7f630d14bb4a6ff4489577a044c, State: Running, Role: LEADER
I20260928 09:08:18.051738 18395 consensus_queue.cc:237] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } }
I20260928 09:08:18.052608 17926 catalog_manager.cc:5495] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6563e7f630d14bb4a6ff4489577a044c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6563e7f630d14bb4a6ff4489577a044c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6563e7f630d14bb4a6ff4489577a044c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39769 } health_report { overall_health: HEALTHY } } }
I20260928 09:08:18.089133 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:18.086642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.0.98.1:39769)
0928 09:08:18.086714 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:18.086717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:18.086725 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:18.086775 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:18.086776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:18.086804 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:18.086883 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:18.087029 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.087520 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.088283 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:18.088488 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:18.088567 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:18.088744 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:18.088857 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:18.088948 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:18.088992 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 09:08:18.089013 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:08:18.091058 18199 raft_consensus.cc:1261] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 1 FOLLOWER]: Refusing update from remote peer 6563e7f630d14bb4a6ff4489577a044c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:18.091058 18330 raft_consensus.cc:1261] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 1 FOLLOWER]: Refusing update from remote peer 6563e7f630d14bb4a6ff4489577a044c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:18.091477 18395 consensus_queue.cc:1035] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6db0ff40c54eea8c18ed65db6f7905" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:18.091637 18400 consensus_queue.cc:1035] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1968567fade84fd0a0cb6b77332addc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:18.094153 18404 mvcc.cc:205] Tried to move back new op lower bound from 7334242296177975296 to 7334242296020475904. Current Snapshot: MvccSnapshot[applied={T|T < 7334242296177975296}]
I20260928 09:08:18.094626 18406 mvcc.cc:205] Tried to move back new op lower bound from 7334242296177975296 to 7334242296020475904. Current Snapshot: MvccSnapshot[applied={T|T < 7334242296177975296}]
I20260928 09:08:18.096217 17926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:18.096324 17926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:18.097100 17926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f61fbcb221154d018e1547fea72a6cf3: Sending DeleteTablet for 3 replicas of tablet 577be77dd3ca42d281a821a456b5f97a
I20260928 09:08:18.097427 18048 tablet_service.cc:1514] Processing DeleteTablet for tablet 577be77dd3ca42d281a821a456b5f97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:18 UTC) from {username='slave'} at 127.0.0.1:43000
I20260928 09:08:18.097486 18179 tablet_service.cc:1514] Processing DeleteTablet for tablet 577be77dd3ca42d281a821a456b5f97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:18 UTC) from {username='slave'} at 127.0.0.1:36570
I20260928 09:08:18.097568 18310 tablet_service.cc:1514] Processing DeleteTablet for tablet 577be77dd3ca42d281a821a456b5f97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:18 UTC) from {username='slave'} at 127.0.0.1:58690
I20260928 09:08:18.097705 18414 tablet_replica.cc:330] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: stopping tablet replica
I20260928 09:08:18.097801 18414 raft_consensus.cc:2229] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:18.097911 18414 raft_consensus.cc:2258] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:18.097915 18413 tablet_replica.cc:330] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c: stopping tablet replica
I20260928 09:08:18.097997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17986
I20260928 09:08:18.098018 18413 raft_consensus.cc:2229] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:08:18.098152 18413 raft_consensus.cc:2258] T 577be77dd3ca42d281a821a456b5f97a P 6563e7f630d14bb4a6ff4489577a044c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:18.098373 18414 ts_tablet_manager.cc:1905] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:18.100095 18415 tablet_replica.cc:330] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: stopping tablet replica
I20260928 09:08:18.100226 18415 raft_consensus.cc:2229] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:18.100366 18415 raft_consensus.cc:2258] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:18.100754 18414 ts_tablet_manager.cc:1918] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:18.100833 18414 log.cc:1196] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/577be77dd3ca42d281a821a456b5f97a
I20260928 09:08:18.100822 18415 ts_tablet_manager.cc:1905] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:18.101105 18414 ts_tablet_manager.cc:1939] T 577be77dd3ca42d281a821a456b5f97a P 3a6db0ff40c54eea8c18ed65db6f7905: Deleting consensus metadata
I20260928 09:08:18.101570 17911 catalog_manager.cc:4841] TS 3a6db0ff40c54eea8c18ed65db6f7905 (127.0.98.2:39343): tablet 577be77dd3ca42d281a821a456b5f97a (table test-table [id=d0bfd3b1bfda4800985f1e2e42824dd4]) successfully deleted
I20260928 09:08:18.102437 18415 ts_tablet_manager.cc:1918] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:18.102519 18415 log.cc:1196] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/577be77dd3ca42d281a821a456b5f97a
I20260928 09:08:18.102756 18415 ts_tablet_manager.cc:1939] T 577be77dd3ca42d281a821a456b5f97a P 1968567fade84fd0a0cb6b77332addc7: Deleting consensus metadata
I20260928 09:08:18.103229 17911 catalog_manager.cc:4841] TS 1968567fade84fd0a0cb6b77332addc7 (127.0.98.3:33731): tablet 577be77dd3ca42d281a821a456b5f97a (table test-table [id=d0bfd3b1bfda4800985f1e2e42824dd4]) successfully deleted
W20260928 09:08:18.104141 17912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39769 (error 108)
I20260928 09:08:18.104665 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:18.104525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39769 (local address 127.0.0.1:43014)
0928 09:08:18.104571 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:18.104611 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 09:08:18.104750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18117
W20260928 09:08:18.104820 17912 catalog_manager.cc:4584] TS 6563e7f630d14bb4a6ff4489577a044c (127.0.98.1:39769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 577be77dd3ca42d281a821a456b5f97a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39769: connect: Connection refused (error 111)
I20260928 09:08:18.109756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18248
I20260928 09:08:18.114521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 17895
2026-09-28T09:08:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (828 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:18 dist-test-slave-smns krb5kdc[18419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:18 dist-test-slave-smns krb5kdc[18419](info): set up 2 sockets
Sep 28 09:08:18 dist-test-slave-smns krb5kdc[18419](info): commencing operation
krb5kdc: starting...
W20260928 09:08:19.157193 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:19.276733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34851
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:19.357579 18432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:19.357802 18432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:19.357825 18432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:19.359043 18432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:19.359088 18432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:19.359109 18432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:19.359124 18432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:19.359136 18432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:19.360646 18432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:19.360908 18432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:19.361150 18432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:19.366024 18437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.366115 18432 server_base.cc:1034] running on GCE node
W20260928 09:08:19.366024 18440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:19.366025 18438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.366524 18432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:19.366827 18432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:19.367970 18432 hybrid_clock.cc:648] HybridClock initialized: now 1790586499367949 us; error 32 us; skew 500 ppm
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:19.370711 18432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:19.371225 18432 webserver.cc:460] Webserver started at http://127.0.98.62:35153/ using document root <none> and password file <none>
I20260928 09:08:19.371456 18432 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:19.371503 18432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:19.371636 18432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:19.372727 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "7575e9e582b34f9d8dd5da06f60b0cd4"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "7fa9a3592dbc741680579b19524fd78c"
server_key_iv: "9a5461e22b5a822146f9a6e5420815f6"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.373090 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "7575e9e582b34f9d8dd5da06f60b0cd4"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "7fa9a3592dbc741680579b19524fd78c"
server_key_iv: "9a5461e22b5a822146f9a6e5420815f6"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.374533 18432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:19.375257 18447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.375433 18432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:08:19.375521 18432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "7575e9e582b34f9d8dd5da06f60b0cd4"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "7fa9a3592dbc741680579b19524fd78c"
server_key_iv: "9a5461e22b5a822146f9a6e5420815f6"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.375591 18432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:19.391213 18432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:19.393652 18432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:19.393810 18432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:19.397919 18432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34851
I20260928 09:08:19.397974 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34851 every 8 connection(s)
I20260928 09:08:19.398337 18432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:19.399217 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:19.401399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18432
I20260928 09:08:19.401491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:19.401718 392 external_mini_cluster.cc:1428] Setting key 5583897307965e3caa7db1337865fda6
I20260928 09:08:19.401778 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4: Bootstrap starting.
I20260928 09:08:19.402508 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:19.402814 18500 log.cc:824] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:19.403520 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4: No bootstrap required, opened a new log
I20260928 09:08:19.405404 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260928 09:08:19.405634 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:19.405679 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7575e9e582b34f9d8dd5da06f60b0cd4, State: Initialized, Role: FOLLOWER
I20260928 09:08:19.405797 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260928 09:08:19.405861 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:19.405886 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:19.405929 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:19.406469 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260928 09:08:19.406553 18500 leader_election.cc:302] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7575e9e582b34f9d8dd5da06f60b0cd4; no voters:
I20260928 09:08:19.406755 18500 leader_election.cc:288] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:19.406778 18505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:19.406890 18505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7575e9e582b34f9d8dd5da06f60b0cd4, State: Running, Role: LEADER
I20260928 09:08:19.406960 18505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260928 09:08:19.407016 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:19.407279 18507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7575e9e582b34f9d8dd5da06f60b0cd4. Latest consensus state: current_term: 1 leader_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } } }
I20260928 09:08:19.407387 18507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:19.408159 18509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:19.408784 18509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:19.409385 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7575e9e582b34f9d8dd5da06f60b0cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } } }
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:19.409902 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:19.410518 18509 catalog_manager.cc:1347] Generated new cluster ID: 9ced0d921b014971a1c40a1387fe0871
I20260928 09:08:19.410570 18509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:19.414106 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.402244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.0.98.62:34851)
0928 09:08:19.402445 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.402454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.402489 (+ 35us) server_negotiation.cc:407] Connection header received
0928 09:08:19.403103 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.403110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.403158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.403307 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:19.403982 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.405980 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.406828 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.407411 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.407535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.410446 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:19.410450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:19.410450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:19.410461 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:19.412362 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.412805 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.412807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.412807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.412904 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.413117 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.413118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.413118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.413398 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:19.413533 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.413782 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.413909 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260928 09:08:19.420147 18509 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:19.420635 18509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:19.424841 18509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4: Generated new TSK 0
I20260928 09:08:19.425019 18509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:08:19.455958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34851
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:19.537994 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:19.538208 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:19.538232 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:19.539412 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:19.539454 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:19.539480 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:08:19.540983 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:19.541244 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:19.541476 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:19.546741 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.546842 18526 server_base.cc:1034] running on GCE node
W20260928 09:08:19.546785 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:19.546785 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.547252 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:19.547464 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:19.548641 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790586499548617 us; error 36 us; skew 500 ppm
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:19.551470 18526 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:08:19.551949 18526 webserver.cc:460] Webserver started at http://127.0.98.1:40113/ using document root <none> and password file <none>
I20260928 09:08:19.552208 18526 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:19.552278 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:19.552426 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:19.553602 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "754ebd6b927b4270b2f1736c7c5e444a"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "e921e7125673edbb4df53e5844654545"
server_key_iv: "404e47d0de7359849ed31a1431e66bab"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.553995 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "754ebd6b927b4270b2f1736c7c5e444a"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "e921e7125673edbb4df53e5844654545"
server_key_iv: "404e47d0de7359849ed31a1431e66bab"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.555380 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:19.556134 18541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.556327 18526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:19.556427 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "754ebd6b927b4270b2f1736c7c5e444a"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "e921e7125673edbb4df53e5844654545"
server_key_iv: "404e47d0de7359849ed31a1431e66bab"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.556511 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:19.565452 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:19.567245 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:19.567400 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:19.567615 18526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:19.567916 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:19.567968 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.568009 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:19.568037 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.573902 18526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41349
I20260928 09:08:19.573942 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41349 every 8 connection(s)
I20260928 09:08:19.574288 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:19.580682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18526
I20260928 09:08:19.580798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:08:19.581008 392 external_mini_cluster.cc:1428] Setting key c30bcd387c59c79167df14726e4f6f6f
I20260928 09:08:19.584061 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.575050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49989 (local address 127.0.98.62:34851)
0928 09:08:19.575116 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.575118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.575304 (+ 186us) server_negotiation.cc:407] Connection header received
0928 09:08:19.576219 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.576221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.576230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.576303 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:19.576919 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.577408 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.578501 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.578714 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.578771 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.581043 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:19.581045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:19.581045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:19.581052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:19.582549 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.583100 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.583101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.583102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.583159 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.583447 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.583447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.583447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.583540 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:19.583586 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.583924 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.583996 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 09:08:19.584357 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.575016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34851 (local address 127.0.98.1:49989)
0928 09:08:19.575175 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.575215 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.576124 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.576375 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.576378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.576629 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:19.576843 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.576848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.577481 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.577482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.578430 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.578433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.578790 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.578791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.578892 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.579474 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:19.579481 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:19.580949 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:19.582654 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:19.582662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:19.582664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:19.583030 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:19.583231 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:19.583232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:19.583233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:19.583381 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:19.583598 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:19.583603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:19.583776 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.583904 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.584084 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 09:08:19.584890 18655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34851
I20260928 09:08:19.584997 18655 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:19.585198 18655 heartbeater.cc:511] Master 127.0.98.62:34851 requested a full tablet report, sending...
I20260928 09:08:19.585815 18464 ts_manager.cc:194] Registered new tserver with Master: 754ebd6b927b4270b2f1736c7c5e444a (127.0.98.1:41349)
I20260928 09:08:19.586460 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49989
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:08:19.608690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34851
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:19.691045 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:19.691267 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:19.691289 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:19.692456 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:19.692497 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:19.692523 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:19.694031 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:19.694285 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:19.694527 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:19.699486 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:19.699517 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.699684 18660 server_base.cc:1034] running on GCE node
W20260928 09:08:19.699496 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.699958 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:19.700214 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:19.701411 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790586499701398 us; error 32 us; skew 500 ppm
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:19.704043 18660 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:08:19.704453 18660 webserver.cc:460] Webserver started at http://127.0.98.2:35531/ using document root <none> and password file <none>
I20260928 09:08:19.704635 18660 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:19.704684 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:19.704773 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:19.705844 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "61bb45e4c9874f16aa8264d52fec3970"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "cf28ba91306eb291658302550613a165"
server_key_iv: "3b80da5391bea69296197e9cf9a5f465"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.706151 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61bb45e4c9874f16aa8264d52fec3970"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "cf28ba91306eb291658302550613a165"
server_key_iv: "3b80da5391bea69296197e9cf9a5f465"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.707533 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:19.708315 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.708478 18660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:19.708562 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "61bb45e4c9874f16aa8264d52fec3970"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "cf28ba91306eb291658302550613a165"
server_key_iv: "3b80da5391bea69296197e9cf9a5f465"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.708636 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:19.727921 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:19.730275 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:19.730389 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:19.730567 18660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:19.730895 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:19.730954 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.730978 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:19.730990 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.736577 18660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35791
I20260928 09:08:19.736680 18788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35791 every 8 connection(s)
I20260928 09:08:19.736981 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:19.743884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18660
I20260928 09:08:19.744004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:08:19.744201 392 external_mini_cluster.cc:1428] Setting key e50290bb1a4498bb4fa9287f2c398b4f
I20260928 09:08:19.746817 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.737794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42707 (local address 127.0.98.62:34851)
0928 09:08:19.737861 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.737863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.738036 (+ 173us) server_negotiation.cc:407] Connection header received
0928 09:08:19.739003 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.739005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.739017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.739081 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:19.739730 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.740243 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.741259 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.741458 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.741582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.743604 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:19.743605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:19.743606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:19.743614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:19.745253 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.745737 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.745738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.745738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.745802 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.746058 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.746059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.746059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.746220 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:19.746294 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.746672 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.746751 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:08:19.747150 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.737739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34851 (local address 127.0.98.2:42707)
0928 09:08:19.737918 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.737952 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.738887 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.739160 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.739164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.739411 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:19.739603 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.739607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.740342 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.740344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.741166 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.741168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.741592 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.741593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.741719 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.742169 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:19.742177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:19.743505 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:19.745345 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:19.745352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:19.745354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:19.745650 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:19.745868 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:19.745869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:19.745869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:19.745984 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:19.746325 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:19.746333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:19.746509 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.746629 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.746817 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260928 09:08:19.747592 18789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34851
I20260928 09:08:19.747699 18789 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:19.747898 18789 heartbeater.cc:511] Master 127.0.98.62:34851 requested a full tablet report, sending...
I20260928 09:08:19.748260 18464 ts_manager.cc:194] Registered new tserver with Master: 61bb45e4c9874f16aa8264d52fec3970 (127.0.98.2:35791)
I20260928 09:08:19.748643 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42707
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:08:19.769992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34851
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:19.851040 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:19.851330 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:19.851374 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:19.852563 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:19.852633 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:19.852675 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:19.854215 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:19.854485 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:19.854718 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:19.859790 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.859869 18794 server_base.cc:1034] running on GCE node
W20260928 09:08:19.859815 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:19.859790 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:19.860203 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:19.860420 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:19.861588 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790586499861538 us; error 62 us; skew 500 ppm
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:19.864040 18794 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:08:19.864496 18794 webserver.cc:460] Webserver started at http://127.0.98.3:39137/ using document root <none> and password file <none>
I20260928 09:08:19.864763 18794 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:19.864810 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:19.864948 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:19.866062 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "b96872e447344ece849f512a2fcb5c97"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "b5236f4989d762c283a6ccd5aa6cdae9"
server_key_iv: "2fd37568dd11558f83e455ed35e29e26"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.866412 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b96872e447344ece849f512a2fcb5c97"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "b5236f4989d762c283a6ccd5aa6cdae9"
server_key_iv: "2fd37568dd11558f83e455ed35e29e26"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.867749 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:19.868460 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.868647 18794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:19.868762 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "b96872e447344ece849f512a2fcb5c97"
format_stamp: "Formatted at 2026-09-28 09:08:19 on dist-test-slave-smns"
server_key: "b5236f4989d762c283a6ccd5aa6cdae9"
server_key_iv: "2fd37568dd11558f83e455ed35e29e26"
server_key_version: "encryptionkey@0"
I20260928 09:08:19.868832 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:19.880136 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:19.881994 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:19.882140 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:19.882344 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:19.882637 18794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:19.882678 18794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.882717 18794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:19.882747 18794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:19.888576 18794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34581
I20260928 09:08:19.888618 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34581 every 8 connection(s)
I20260928 09:08:19.888913 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:08:19.894322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18794
I20260928 09:08:19.894452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:08:19.894686 392 external_mini_cluster.cc:1428] Setting key 9f094563a3fd48e8a98ce6ff8046f0c3
Sep 28 09:08:19 dist-test-slave-smns krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:19.898630 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.889661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59489 (local address 127.0.98.62:34851)
0928 09:08:19.889706 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.889708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.889905 (+ 197us) server_negotiation.cc:407] Connection header received
0928 09:08:19.890990 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.890992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.891006 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.891060 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:19.891705 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.892213 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.893250 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.893452 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.893575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.896147 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:19.896149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:19.896149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:19.896156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:19.897416 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.897847 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.897848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.897848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.897899 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.898091 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.898092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.898092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.898220 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:19.898270 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.898504 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.898567 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:08:19.898792 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.889622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34851 (local address 127.0.98.3:59489)
0928 09:08:19.889775 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.889819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.890853 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.891140 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.891144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.891407 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:19.891604 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.891609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.892308 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.892309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.893150 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.893152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.893566 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.893568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.893708 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.894289 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:19.894298 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:19.896061 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:19.897493 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:19.897498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:19.897500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:19.897784 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:19.897943 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:19.897943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:19.897943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:19.898030 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:19.898278 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:19.898282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:19.898394 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.898474 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.898593 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260928 09:08:19.899341 18923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34851
I20260928 09:08:19.899426 18923 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:19.899580 18923 heartbeater.cc:511] Master 127.0.98.62:34851 requested a full tablet report, sending...
I20260928 09:08:19.899977 18464 ts_manager.cc:194] Registered new tserver with Master: b96872e447344ece849f512a2fcb5c97 (127.0.98.3:34581)
I20260928 09:08:19.900513 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59489
I20260928 09:08:19.906674 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:08:19.914669 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.908225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.0.98.62:34851)
0928 09:08:19.908311 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.908312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.908379 (+ 67us) server_negotiation.cc:407] Connection header received
0928 09:08:19.908451 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.908452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.908463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.908542 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:19.909229 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.909818 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.910554 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.910737 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.910790 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.912075 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:19.912076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:19.912076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:19.912082 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:19.913347 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.913759 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.913759 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.913760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.913809 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:19.914099 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:19.914099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:19.914100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:19.914247 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:19.914331 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.914466 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.914514 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 09:08:19.916352 18464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:08:19.918066 18464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:08:19.924445 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.920181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35791 (local address 127.0.0.1:34190)
0928 09:08:19.920657 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.920691 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.920741 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.921004 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.921005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.921007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:19.921047 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.921049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.922296 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.922297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.923303 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.923306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.924154 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.924154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.924237 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.924248 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.924306 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.924366 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260928 09:08:19.924449 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.920430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.2:35791)
0928 09:08:19.920615 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.920621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.920796 (+ 175us) server_negotiation.cc:407] Connection header received
0928 09:08:19.920840 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.920842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.920855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.920922 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:19.921136 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.922185 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.923389 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.924070 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.924150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.924209 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.924320 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.924368 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260928 09:08:19.924691 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.920172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41349 (local address 127.0.0.1:41386)
0928 09:08:19.920430 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.920470 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.920538 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.920761 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.920764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.920767 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:19.920843 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.920845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.922141 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.922142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.923529 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.923530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.924387 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.924388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.924451 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.924454 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.924613 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.924643 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":130,"thread_start_us":73,"threads_started":1}
I20260928 09:08:19.924984 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.920365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.98.1:41349)
0928 09:08:19.920530 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.920535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.920549 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:08:19.920635 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.920637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.920652 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.920757 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:19.920927 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.922037 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.923614 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.924307 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.924377 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.924428 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.924884 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.924928 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260928 09:08:19.925171 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca327fee746432d971b350215c3c899 (DEFAULT_TABLE table=test-table [id=59561877904e4b5db692a960872d621e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:19.925570 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca327fee746432d971b350215c3c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:19.925618 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca327fee746432d971b350215c3c899 (DEFAULT_TABLE table=test-table [id=59561877904e4b5db692a960872d621e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:19.925947 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca327fee746432d971b350215c3c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:19.925985 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.920181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34581 (local address 127.0.0.1:51858)
0928 09:08:19.920524 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.920535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.920589 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.921211 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.921212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.921213 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:19.921248 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.921249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.922185 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.922186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.923528 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.923530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.925744 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.925746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.925807 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.925811 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.925862 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.925908 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":91,"thread_start_us":62,"threads_started":1}
I20260928 09:08:19.926266 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.920335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.0.98.3:34581)
0928 09:08:19.920708 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.920712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.920722 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:08:19.920761 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.920764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.920780 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.920853 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:19.921341 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.922055 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.924466 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.925530 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.925639 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.925697 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.926140 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.926184 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":82,"threads_started":1}
I20260928 09:08:19.926841 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca327fee746432d971b350215c3c899 (DEFAULT_TABLE table=test-table [id=59561877904e4b5db692a960872d621e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:19.927173 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca327fee746432d971b350215c3c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:19.928552 18937 tablet_bootstrap.cc:492] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: Bootstrap starting.
I20260928 09:08:19.928952 18938 tablet_bootstrap.cc:492] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Bootstrap starting.
I20260928 09:08:19.929352 18937 tablet_bootstrap.cc:654] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:19.929636 18938 tablet_bootstrap.cc:654] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:19.930022 18937 log.cc:824] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:19.930198 18938 log.cc:824] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:19.930405 18939 tablet_bootstrap.cc:492] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Bootstrap starting.
I20260928 09:08:19.930922 18938 tablet_bootstrap.cc:492] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: No bootstrap required, opened a new log
I20260928 09:08:19.930959 18939 tablet_bootstrap.cc:654] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:19.931005 18938 ts_tablet_manager.cc:1397] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:08:19.931444 18939 log.cc:824] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:19.932150 18937 tablet_bootstrap.cc:492] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: No bootstrap required, opened a new log
I20260928 09:08:19.932188 18939 tablet_bootstrap.cc:492] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: No bootstrap required, opened a new log
I20260928 09:08:19.932224 18937 ts_tablet_manager.cc:1397] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 09:08:19.932263 18939 ts_tablet_manager.cc:1397] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:19.932922 18938 raft_consensus.cc:348] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.933110 18938 raft_consensus.cc:374] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:19.933147 18938 raft_consensus.cc:729] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61bb45e4c9874f16aa8264d52fec3970, State: Initialized, Role: FOLLOWER
I20260928 09:08:19.933285 18938 consensus_queue.cc:260] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.933581 18938 ts_tablet_manager.cc:1428] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:08:19.933729 18789 heartbeater.cc:503] Master 127.0.98.62:34851 was elected leader, sending a full tablet report...
I20260928 09:08:19.933921 18937 raft_consensus.cc:348] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.934103 18937 raft_consensus.cc:374] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:19.934151 18937 raft_consensus.cc:729] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754ebd6b927b4270b2f1736c7c5e444a, State: Initialized, Role: FOLLOWER
I20260928 09:08:19.934259 18937 consensus_queue.cc:260] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.934291 18939 raft_consensus.cc:348] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.934478 18939 raft_consensus.cc:374] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:19.934525 18939 raft_consensus.cc:729] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96872e447344ece849f512a2fcb5c97, State: Initialized, Role: FOLLOWER
I20260928 09:08:19.934535 18937 ts_tablet_manager.cc:1428] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 09:08:19.934666 18655 heartbeater.cc:503] Master 127.0.98.62:34851 was elected leader, sending a full tablet report...
I20260928 09:08:19.934664 18939 consensus_queue.cc:260] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.934958 18939 ts_tablet_manager.cc:1428] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:19.935040 18923 heartbeater.cc:503] Master 127.0.98.62:34851 was elected leader, sending a full tablet report...
I20260928 09:08:19.958940 18944 raft_consensus.cc:482] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:19.959052 18944 raft_consensus.cc:504] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.959547 18944 leader_election.cc:288] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b96872e447344ece849f512a2fcb5c97 (127.0.98.3:34581), 61bb45e4c9874f16aa8264d52fec3970 (127.0.98.2:35791)
I20260928 09:08:19.962473 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.959708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35791 (local address 127.0.98.1:55505)
0928 09:08:19.959974 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.959984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.960033 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.960173 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.960174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.960176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:19.960248 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.960250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.960882 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.960884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.961653 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.961656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.962210 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.962212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.962289 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.962294 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.962346 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.962383 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":86,"thread_start_us":112,"threads_started":1}
I20260928 09:08:19.962489 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.959898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55505 (local address 127.0.98.2:35791)
0928 09:08:19.959945 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.959947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.960068 (+ 121us) server_negotiation.cc:407] Connection header received
0928 09:08:19.960098 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.960099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.960112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.960164 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:19.960317 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.960772 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.961755 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.962125 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.962180 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.962188 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.962368 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.962410 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:08:19.962612 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.959823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43413 (local address 127.0.98.3:34581)
0928 09:08:19.959881 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.959884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.959892 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:19.959919 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.959920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.959930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.959984 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:19.960230 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.960742 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.961810 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.962202 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.962273 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.962284 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.962482 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.962530 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 09:08:19.962563 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.959708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34581 (local address 127.0.98.1:43413)
0928 09:08:19.959773 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.959787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:19.959839 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:19.960012 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:19.960013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:19.960033 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:19.960106 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.960109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.960825 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.960827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.961690 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:19.961692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.962290 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.962291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:19.962397 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.962400 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:19.962458 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:08:19.962493 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 09:08:19.963119 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca327fee746432d971b350215c3c899" candidate_uuid: "754ebd6b927b4270b2f1736c7c5e444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61bb45e4c9874f16aa8264d52fec3970" is_pre_election: true
I20260928 09:08:19.963235 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca327fee746432d971b350215c3c899" candidate_uuid: "754ebd6b927b4270b2f1736c7c5e444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96872e447344ece849f512a2fcb5c97" is_pre_election: true
I20260928 09:08:19.963326 18743 raft_consensus.cc:2454] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754ebd6b927b4270b2f1736c7c5e444a in term 0.
I20260928 09:08:19.963407 18877 raft_consensus.cc:2454] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754ebd6b927b4270b2f1736c7c5e444a in term 0.
I20260928 09:08:19.963575 18545 leader_election.cc:302] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61bb45e4c9874f16aa8264d52fec3970, 754ebd6b927b4270b2f1736c7c5e444a; no voters:
I20260928 09:08:19.963738 18944 raft_consensus.cc:2781] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:08:19.963794 18944 raft_consensus.cc:482] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:08:19.963820 18944 raft_consensus.cc:3037] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:19.964449 18944 raft_consensus.cc:504] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.964596 18944 leader_election.cc:288] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [CANDIDATE]: Term 1 election: Requested vote from peers b96872e447344ece849f512a2fcb5c97 (127.0.98.3:34581), 61bb45e4c9874f16aa8264d52fec3970 (127.0.98.2:35791)
I20260928 09:08:19.964759 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca327fee746432d971b350215c3c899" candidate_uuid: "754ebd6b927b4270b2f1736c7c5e444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61bb45e4c9874f16aa8264d52fec3970"
I20260928 09:08:19.964771 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca327fee746432d971b350215c3c899" candidate_uuid: "754ebd6b927b4270b2f1736c7c5e444a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96872e447344ece849f512a2fcb5c97"
I20260928 09:08:19.964844 18743 raft_consensus.cc:3037] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:19.964845 18877 raft_consensus.cc:3037] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:19.965771 18877 raft_consensus.cc:2454] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754ebd6b927b4270b2f1736c7c5e444a in term 1.
I20260928 09:08:19.965771 18743 raft_consensus.cc:2454] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754ebd6b927b4270b2f1736c7c5e444a in term 1.
I20260928 09:08:19.965953 18545 leader_election.cc:302] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61bb45e4c9874f16aa8264d52fec3970, 754ebd6b927b4270b2f1736c7c5e444a; no voters:
I20260928 09:08:19.966055 18944 raft_consensus.cc:2781] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:19.966229 18944 raft_consensus.cc:686] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 1 LEADER]: Becoming Leader. State: Replica: 754ebd6b927b4270b2f1736c7c5e444a, State: Running, Role: LEADER
I20260928 09:08:19.966338 18944 consensus_queue.cc:237] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } }
I20260928 09:08:19.967123 18463 catalog_manager.cc:5495] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a reported cstate change: term changed from 0 to 1, leader changed from <none> to 754ebd6b927b4270b2f1736c7c5e444a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "754ebd6b927b4270b2f1736c7c5e444a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754ebd6b927b4270b2f1736c7c5e444a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 } health_report { overall_health: UNKNOWN } } }
I20260928 09:08:19.973276 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.971092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.0.98.1:41349)
0928 09:08:19.971146 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:19.971149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:19.971157 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:19.971193 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:19.971193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:19.971209 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:19.971271 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:19.971389 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.971829 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.972638 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:19.972834 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:19.972891 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:19.973018 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:19.973097 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:19.973167 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:19.973206 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 09:08:19.973215 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:08:19.974985 18877 raft_consensus.cc:1261] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 1 FOLLOWER]: Refusing update from remote peer 754ebd6b927b4270b2f1736c7c5e444a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:19.974985 18743 raft_consensus.cc:1261] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 1 FOLLOWER]: Refusing update from remote peer 754ebd6b927b4270b2f1736c7c5e444a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:19.975303 18947 consensus_queue.cc:1035] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b96872e447344ece849f512a2fcb5c97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:19.975421 18944 consensus_queue.cc:1035] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61bb45e4c9874f16aa8264d52fec3970" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:19.980821 18464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:19.980927 18464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:19.981748 18464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7575e9e582b34f9d8dd5da06f60b0cd4: Sending DeleteTablet for 3 replicas of tablet 5ca327fee746432d971b350215c3c899
I20260928 09:08:19.982089 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca327fee746432d971b350215c3c899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41386
I20260928 09:08:19.982105 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca327fee746432d971b350215c3c899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34190
I20260928 09:08:19.982147 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca327fee746432d971b350215c3c899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51858
I20260928 09:08:19.982331 18960 tablet_replica.cc:330] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a: stopping tablet replica
I20260928 09:08:19.982429 18960 raft_consensus.cc:2229] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:08:19.982553 18960 raft_consensus.cc:2258] T 5ca327fee746432d971b350215c3c899 P 754ebd6b927b4270b2f1736c7c5e444a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:19.982596 18962 tablet_replica.cc:330] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: stopping tablet replica
I20260928 09:08:19.982671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18526
I20260928 09:08:19.984212 18962 raft_consensus.cc:2229] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:19.984366 18962 raft_consensus.cc:2258] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:19.984659 18961 tablet_replica.cc:330] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: stopping tablet replica
I20260928 09:08:19.984791 18961 raft_consensus.cc:2229] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:19.984808 18962 ts_tablet_manager.cc:1905] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:19.984900 18961 raft_consensus.cc:2258] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:19.985311 18961 ts_tablet_manager.cc:1905] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:19.986907 18961 ts_tablet_manager.cc:1918] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:19.986999 18961 log.cc:1196] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/5ca327fee746432d971b350215c3c899
I20260928 09:08:19.987185 18962 ts_tablet_manager.cc:1918] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:19.987267 18962 log.cc:1196] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/5ca327fee746432d971b350215c3c899
I20260928 09:08:19.987304 18961 ts_tablet_manager.cc:1939] T 5ca327fee746432d971b350215c3c899 P b96872e447344ece849f512a2fcb5c97: Deleting consensus metadata
I20260928 09:08:19.987581 18962 ts_tablet_manager.cc:1939] T 5ca327fee746432d971b350215c3c899 P 61bb45e4c9874f16aa8264d52fec3970: Deleting consensus metadata
I20260928 09:08:19.987720 18448 catalog_manager.cc:4841] TS b96872e447344ece849f512a2fcb5c97 (127.0.98.3:34581): tablet 5ca327fee746432d971b350215c3c899 (table test-table [id=59561877904e4b5db692a960872d621e]) successfully deleted
I20260928 09:08:19.987996 18451 catalog_manager.cc:4841] TS 61bb45e4c9874f16aa8264d52fec3970 (127.0.98.2:35791): tablet 5ca327fee746432d971b350215c3c899 (table test-table [id=59561877904e4b5db692a960872d621e]) successfully deleted
W20260928 09:08:19.988736 18449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41349 (error 108)
I20260928 09:08:19.988945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18660
I20260928 09:08:19.989630 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:19.989076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41349 (local address 127.0.0.1:41412)
0928 09:08:19.989533 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:19.989592 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
W20260928 09:08:19.989995 18449 catalog_manager.cc:4584] TS 754ebd6b927b4270b2f1736c7c5e444a (127.0.98.1:41349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ca327fee746432d971b350215c3c899: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41349: connect: Connection refused (error 111)
I20260928 09:08:19.993726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18794
I20260928 09:08:19.999123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18432
2026-09-28T09:08:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1885 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T09:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:20Z Disabled control of system clock
I20260928 09:08:20.026821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41529
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:20.107491 18967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:20.107733 18967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:20.107755 18967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:20.108975 18967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:20.109050 18967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:20.109073 18967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:20.109112 18967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:20.109164 18967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:20.110759 18967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:20.110986 18967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:20.111241 18967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:20.116381 18972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:20.116389 18973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.116420 18967 server_base.cc:1034] running on GCE node
W20260928 09:08:20.116389 18975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.116990 18967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:20.117295 18967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:20.118463 18967 hybrid_clock.cc:648] HybridClock initialized: now 1790586500118453 us; error 40 us; skew 500 ppm
I20260928 09:08:20.119788 18967 webserver.cc:460] Webserver started at http://127.0.98.62:45343/ using document root <none> and password file <none>
I20260928 09:08:20.120040 18967 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:20.120111 18967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:20.120270 18967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:20.121376 18967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "4877badc20ba49188d4f846f4f2f4e73"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "6c0500465fd824370464e126f475014f"
server_key_iv: "84e9ad4a64f627af5e89e1838bbe3240"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.121767 18967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "4877badc20ba49188d4f846f4f2f4e73"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "6c0500465fd824370464e126f475014f"
server_key_iv: "84e9ad4a64f627af5e89e1838bbe3240"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.123150 18967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:20.123956 18981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.124135 18967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:20.124260 18967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "4877badc20ba49188d4f846f4f2f4e73"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "6c0500465fd824370464e126f475014f"
server_key_iv: "84e9ad4a64f627af5e89e1838bbe3240"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.124344 18967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:20.137301 18967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:20.137655 18967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:20.137806 18967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:20.141400 18967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41529
I20260928 09:08:20.141460 19033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41529 every 8 connection(s)
I20260928 09:08:20.141836 18967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:20.142776 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:20.145054 19034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73: Bootstrap starting.
I20260928 09:08:20.145712 19034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:20.145962 19034 log.cc:824] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:20.146661 19034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73: No bootstrap required, opened a new log
I20260928 09:08:20.148290 19034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877badc20ba49188d4f846f4f2f4e73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41529 } }
I20260928 09:08:20.148432 19034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:20.148454 19034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4877badc20ba49188d4f846f4f2f4e73, State: Initialized, Role: FOLLOWER
I20260928 09:08:20.148631 19034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877badc20ba49188d4f846f4f2f4e73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41529 } }
I20260928 09:08:20.148720 19034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:20.148765 19034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:20.148813 19034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:20.149346 19034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877badc20ba49188d4f846f4f2f4e73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41529 } }
I20260928 09:08:20.149466 19034 leader_election.cc:302] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4877badc20ba49188d4f846f4f2f4e73; no voters:
I20260928 09:08:20.149709 19034 leader_election.cc:288] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:20.149744 19037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:20.149880 19037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [term 1 LEADER]: Becoming Leader. State: Replica: 4877badc20ba49188d4f846f4f2f4e73, State: Running, Role: LEADER
I20260928 09:08:20.149994 19034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:20.149972 19037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877badc20ba49188d4f846f4f2f4e73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41529 } }
I20260928 09:08:20.150388 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4877badc20ba49188d4f846f4f2f4e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877badc20ba49188d4f846f4f2f4e73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41529 } } }
I20260928 09:08:20.150486 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:20.150578 19039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4877badc20ba49188d4f846f4f2f4e73. Latest consensus state: current_term: 1 leader_uuid: "4877badc20ba49188d4f846f4f2f4e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877badc20ba49188d4f846f4f2f4e73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41529 } } }
I20260928 09:08:20.150715 19039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:20.150729 19044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:20.151327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 18967
I20260928 09:08:20.151440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:20.151508 19044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:20.151659 392 external_mini_cluster.cc:1428] Setting key 462f2a6c75f20e1d2e4ecb0cde5f2b65
I20260928 09:08:20.153257 19044 catalog_manager.cc:1347] Generated new cluster ID: b73d245386854d7581be752be6b08521
I20260928 09:08:20.153317 19044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:20.158211 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.152321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.0.98.62:41529)
0928 09:08:20.152744 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.152752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.152789 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:08:20.152915 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.152920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.152973 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.153112 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:20.153359 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.154769 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.155873 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.156491 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.156915 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.157352 (+ 437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:20.157356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:20.157356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:20.157569 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:20.157608 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:20.157621 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:20.157892 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.157921 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.157995 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":73,"threads_started":1}
I20260928 09:08:20.189311 19044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:20.189946 19044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:20.199833 19044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73: Generated new TSK 0
I20260928 09:08:20.200033 19044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:20.206674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41529
--builtin_ntp_servers=127.0.98.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:20.287513 19058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:20.287724 19058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:20.287747 19058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:20.288909 19058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:20.288947 19058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:20.288972 19058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:08:20.290402 19058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:20.290658 19058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:20.290964 19058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:20.295933 19066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:20.295926 19063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.295964 19058 server_base.cc:1034] running on GCE node
W20260928 09:08:20.295931 19064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.296334 19058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:20.296542 19058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:20.297676 19058 hybrid_clock.cc:648] HybridClock initialized: now 1790586500297656 us; error 37 us; skew 500 ppm
I20260928 09:08:20.299063 19058 webserver.cc:460] Webserver started at http://127.0.98.1:37383/ using document root <none> and password file <none>
I20260928 09:08:20.299319 19058 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:20.299368 19058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:20.299535 19058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:20.300626 19058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "068a3ee55db74630a8b3f8bb0e449b42"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "3b745a1df1eb136c4b9bac877cf1b5ff"
server_key_iv: "cd3ce8b27cad0e915b3f3b50c3143c5d"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.300992 19058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "068a3ee55db74630a8b3f8bb0e449b42"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "3b745a1df1eb136c4b9bac877cf1b5ff"
server_key_iv: "cd3ce8b27cad0e915b3f3b50c3143c5d"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.302419 19058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:20.303241 19072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.303433 19058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:20.303539 19058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "068a3ee55db74630a8b3f8bb0e449b42"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "3b745a1df1eb136c4b9bac877cf1b5ff"
server_key_iv: "cd3ce8b27cad0e915b3f3b50c3143c5d"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.303625 19058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:20.334797 19058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:20.335160 19058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:20.335325 19058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:20.335580 19058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:20.335948 19058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:20.336004 19058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.336077 19058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:20.336108 19058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.342149 19058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42525
I20260928 09:08:20.342202 19185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42525 every 8 connection(s)
I20260928 09:08:20.342568 19058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:08:20.348129 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.343353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49749 (local address 127.0.98.62:41529)
0928 09:08:20.343426 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.343429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.343817 (+ 388us) server_negotiation.cc:407] Connection header received
0928 09:08:20.344297 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.344298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.344312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.344373 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:20.345071 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.345676 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.346714 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.346914 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.346974 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.347902 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:20.347903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:20.347904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:20.347908 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:20.347923 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:20.347927 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:20.347972 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.348040 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.348065 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:08:20.348304 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.343357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41529 (local address 127.0.98.1:49749)
0928 09:08:20.343653 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.343698 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.344196 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.344449 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.344452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.344461 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:20.344975 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.344979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.345782 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.345783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.346608 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.346611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.346994 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.346994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.347117 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.347762 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:20.347763 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:20.347793 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:20.347817 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:20.347972 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:20.347975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:20.347977 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.348026 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.348107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260928 09:08:20.348824 19186 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41529
I20260928 09:08:20.348976 19186 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:20.349150 19186 heartbeater.cc:511] Master 127.0.98.62:41529 requested a full tablet report, sending...
I20260928 09:08:20.349777 18998 ts_manager.cc:194] Registered new tserver with Master: 068a3ee55db74630a8b3f8bb0e449b42 (127.0.98.1:42525)
I20260928 09:08:20.350497 18998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49749
I20260928 09:08:20.352146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19058
I20260928 09:08:20.352242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:08:20.352406 392 external_mini_cluster.cc:1428] Setting key 115e7037dbc1394661b186ad56db9fd5
I20260928 09:08:20.353338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41529
--builtin_ntp_servers=127.0.98.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:20.435946 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:20.436269 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:20.436295 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:20.437474 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:20.437512 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:20.437537 19189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:20.438956 19189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:20.439215 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:20.439510 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:20.444483 19195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:20.444516 19194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:20.444495 19197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.444785 19189 server_base.cc:1034] running on GCE node
I20260928 09:08:20.445010 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:20.445253 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:20.446416 19189 hybrid_clock.cc:648] HybridClock initialized: now 1790586500446396 us; error 38 us; skew 500 ppm
I20260928 09:08:20.447762 19189 webserver.cc:460] Webserver started at http://127.0.98.2:39955/ using document root <none> and password file <none>
I20260928 09:08:20.447999 19189 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:20.448045 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:20.448218 19189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:20.449378 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "722635a88f3245c6948359347582d082"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "8335c7b302bda457ab45869614400352"
server_key_iv: "c6b17aa917d091a81e99b8f1291465a0"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.449875 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "722635a88f3245c6948359347582d082"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "8335c7b302bda457ab45869614400352"
server_key_iv: "c6b17aa917d091a81e99b8f1291465a0"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.451442 19189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:20.452314 19203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.452518 19189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:20.452610 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "722635a88f3245c6948359347582d082"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "8335c7b302bda457ab45869614400352"
server_key_iv: "c6b17aa917d091a81e99b8f1291465a0"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.452682 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:20.464051 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:20.464358 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:20.464505 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:20.464720 19189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:20.465039 19189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:20.465073 19189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.465095 19189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:20.465149 19189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.471011 19189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34051
I20260928 09:08:20.471052 19316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34051 every 8 connection(s)
I20260928 09:08:20.471376 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:08:20.476707 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.472070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47831 (local address 127.0.98.62:41529)
0928 09:08:20.472126 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.472128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.472360 (+ 232us) server_negotiation.cc:407] Connection header received
0928 09:08:20.472877 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.472878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.472890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.472957 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:20.473760 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.474257 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.475276 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.475483 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.475553 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.476435 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:20.476455 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:20.476455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:20.476460 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:20.476475 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:20.476479 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:20.476529 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.476613 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.476642 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 09:08:20.476951 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.472024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41529 (local address 127.0.98.2:47831)
0928 09:08:20.472221 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.472263 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.472756 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.473036 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.473040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.473048 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:20.473670 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.473675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.474354 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.474355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.475142 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.475144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.475597 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.475598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.475725 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.476289 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:20.476289 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:20.476320 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:20.476343 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:20.476539 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:20.476543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:20.476546 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.476601 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.476690 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260928 09:08:20.477579 19317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41529
I20260928 09:08:20.477677 19317 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:20.477861 19317 heartbeater.cc:511] Master 127.0.98.62:41529 requested a full tablet report, sending...
I20260928 09:08:20.478099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19189
I20260928 09:08:20.478224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:08:20.478319 18998 ts_manager.cc:194] Registered new tserver with Master: 722635a88f3245c6948359347582d082 (127.0.98.2:34051)
I20260928 09:08:20.478408 392 external_mini_cluster.cc:1428] Setting key a91fed9928978e7d816facbc3e6a2978
I20260928 09:08:20.478755 18998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47831
I20260928 09:08:20.479514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41529
--builtin_ntp_servers=127.0.98.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:20.561105 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:20.561358 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:20.561399 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:20.562654 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:20.562726 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:20.562767 19320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:20.564139 19320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:20.564421 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:20.564729 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:20.570052 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:20.570029 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.570061 19320 server_base.cc:1034] running on GCE node
W20260928 09:08:20.570030 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:20.570429 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:20.570636 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:20.571777 19320 hybrid_clock.cc:648] HybridClock initialized: now 1790586500571767 us; error 34 us; skew 500 ppm
I20260928 09:08:20.573071 19320 webserver.cc:460] Webserver started at http://127.0.98.3:39463/ using document root <none> and password file <none>
I20260928 09:08:20.573282 19320 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:20.573323 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:20.573411 19320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:20.574553 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9f938d87f5c4132a7b3f9bf9497ee26"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "f82cd4337370e2c6e6751f57f35e3f54"
server_key_iv: "7b4a5e62a269c8c9feb5e67a2967ceaa"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.574849 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9f938d87f5c4132a7b3f9bf9497ee26"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "f82cd4337370e2c6e6751f57f35e3f54"
server_key_iv: "7b4a5e62a269c8c9feb5e67a2967ceaa"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.576102 19320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:20.576776 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.576936 19320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:20.577024 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "e9f938d87f5c4132a7b3f9bf9497ee26"
format_stamp: "Formatted at 2026-09-28 09:08:20 on dist-test-slave-smns"
server_key: "f82cd4337370e2c6e6751f57f35e3f54"
server_key_iv: "7b4a5e62a269c8c9feb5e67a2967ceaa"
server_key_version: "encryptionkey@0"
I20260928 09:08:20.577106 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:20.599104 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:20.599447 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:20.599607 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:20.599850 19320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:20.600232 19320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:20.600289 19320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.600327 19320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:20.600353 19320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:20.605978 19320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42445
I20260928 09:08:20.606094 19447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42445 every 8 connection(s)
I20260928 09:08:20.606377 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:08:20.611920 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.607222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55509 (local address 127.0.98.62:41529)
0928 09:08:20.607294 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.607297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.607504 (+ 207us) server_negotiation.cc:407] Connection header received
0928 09:08:20.608067 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.608069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.608082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.608132 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:20.608802 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.609304 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.610463 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.610661 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.610767 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.611626 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:20.611627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:20.611628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:20.611632 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:20.611648 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:20.611652 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:20.611723 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.611810 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.611841 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:20.612128 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.607131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41529 (local address 127.0.98.3:55509)
0928 09:08:20.607384 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.607418 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.607950 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.608199 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.608203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.608213 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:20.608710 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.608714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.609399 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.609400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.610372 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.610375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.610777 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.610778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.610917 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.611475 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:20.611475 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:20.611504 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:20.611527 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:20.611725 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:20.611731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:20.611734 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.611808 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.611906 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260928 09:08:20.612638 19448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41529
I20260928 09:08:20.612720 19448 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:20.612859 19448 heartbeater.cc:511] Master 127.0.98.62:41529 requested a full tablet report, sending...
I20260928 09:08:20.613201 18997 ts_manager.cc:194] Registered new tserver with Master: e9f938d87f5c4132a7b3f9bf9497ee26 (127.0.98.3:42445)
I20260928 09:08:20.613667 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55509
I20260928 09:08:20.614298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19320
I20260928 09:08:20.614408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:08:20.614605 392 external_mini_cluster.cc:1428] Setting key d206fe19595ac8eccc5f357dd974157e
I20260928 09:08:20.615621 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:08:20.620252 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.617139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.0.98.62:41529)
0928 09:08:20.617184 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.617186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.617350 (+ 164us) server_negotiation.cc:407] Connection header received
0928 09:08:20.617413 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.617414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.617426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.617479 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:20.617686 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.618215 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.619147 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.619372 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.619427 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.619975 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:20.619976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:20.619976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:20.619980 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:20.619991 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:20.619994 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:20.620061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.620120 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.620130 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:08:20.621794 18997 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:08:20.623297 18997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:08:20.629503 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.625326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42445 (local address 127.0.0.1:50730)
0928 09:08:20.625788 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.625797 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.625841 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.626170 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.626171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.626173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:20.626207 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.626208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.627463 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.627464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.628410 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.628413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.629218 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.629219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.629299 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.629315 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.629363 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.629416 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":4,"thread_start_us":59,"threads_started":1}
I20260928 09:08:20.629956 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.625326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34051 (local address 127.0.0.1:34286)
0928 09:08:20.625498 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.625535 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.625636 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.625894 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.625896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.625898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:20.625966 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.625968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.627297 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.627298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.628925 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.628927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.629721 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.629722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.629784 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.629788 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.629845 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.629885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":83,"thread_start_us":63,"threads_started":1}
I20260928 09:08:20.629989 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.625496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.0.98.2:34051)
0928 09:08:20.625736 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.625742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.625753 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:08:20.625798 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.625800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.625817 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.625888 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:20.626051 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.627204 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.629012 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.629511 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.629632 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.629739 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.629865 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.629905 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
I20260928 09:08:20.630030 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.625494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.0.98.3:42445)
0928 09:08:20.626008 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.626012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.626022 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:08:20.626057 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.626059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.626073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.626278 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:20.626330 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.627379 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.628496 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.629120 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.629757 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.629853 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.629880 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.629918 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":107,"threads_started":1}
I20260928 09:08:20.630322 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.625361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42525 (local address 127.0.0.1:49664)
0928 09:08:20.625897 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.625903 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.625970 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.626328 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.626329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.626331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:20.626373 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.626374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.627385 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.627386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.628707 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.628710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.629884 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.629885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.629943 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.629947 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.630006 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.630236 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":74,"thread_start_us":83,"threads_started":1}
I20260928 09:08:20.630722 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.625630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.0.98.1:42525)
0928 09:08:20.625806 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.625811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.625969 (+ 158us) server_negotiation.cc:407] Connection header received
0928 09:08:20.626148 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.626150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.626178 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.626257 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:20.626467 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.627275 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.628798 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.629568 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.630472 (+ 904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.630569 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.630595 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.630644 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260928 09:08:20.630721 19382 tablet_service.cc:1467] Processing CreateTablet for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 (DEFAULT_TABLE table=test-table [id=d09b169d2a4f4c139788f917b61a6ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:20.630939 19251 tablet_service.cc:1467] Processing CreateTablet for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 (DEFAULT_TABLE table=test-table [id=d09b169d2a4f4c139788f917b61a6ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:20.631098 19382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:20.631287 19251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:20.631315 19120 tablet_service.cc:1467] Processing CreateTablet for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 (DEFAULT_TABLE table=test-table [id=d09b169d2a4f4c139788f917b61a6ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:20.631624 19120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:20.633517 19462 tablet_bootstrap.cc:492] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Bootstrap starting.
I20260928 09:08:20.634124 19462 tablet_bootstrap.cc:654] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:20.634402 19462 log.cc:824] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:20.634696 19463 tablet_bootstrap.cc:492] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Bootstrap starting.
I20260928 09:08:20.635108 19462 tablet_bootstrap.cc:492] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: No bootstrap required, opened a new log
I20260928 09:08:20.635177 19462 ts_tablet_manager.cc:1397] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:08:20.635308 19463 tablet_bootstrap.cc:654] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:20.635602 19463 log.cc:824] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:20.636346 19463 tablet_bootstrap.cc:492] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: No bootstrap required, opened a new log
I20260928 09:08:20.636412 19463 ts_tablet_manager.cc:1397] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:20.637879 19462 raft_consensus.cc:348] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.638048 19462 raft_consensus.cc:374] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:20.638085 19462 raft_consensus.cc:729] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f938d87f5c4132a7b3f9bf9497ee26, State: Initialized, Role: FOLLOWER
I20260928 09:08:20.638010 19465 tablet_bootstrap.cc:492] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Bootstrap starting.
I20260928 09:08:20.638219 19462 consensus_queue.cc:260] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.638482 19462 ts_tablet_manager.cc:1428] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 09:08:20.638589 19448 heartbeater.cc:503] Master 127.0.98.62:41529 was elected leader, sending a full tablet report...
I20260928 09:08:20.638913 19465 tablet_bootstrap.cc:654] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:20.638978 19463 raft_consensus.cc:348] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.639098 19463 raft_consensus.cc:374] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:20.639117 19463 raft_consensus.cc:729] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722635a88f3245c6948359347582d082, State: Initialized, Role: FOLLOWER
I20260928 09:08:20.639225 19465 log.cc:824] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:20.639226 19463 consensus_queue.cc:260] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.639473 19463 ts_tablet_manager.cc:1428] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:20.639585 19317 heartbeater.cc:503] Master 127.0.98.62:41529 was elected leader, sending a full tablet report...
I20260928 09:08:20.640201 19465 tablet_bootstrap.cc:492] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: No bootstrap required, opened a new log
I20260928 09:08:20.640266 19465 ts_tablet_manager.cc:1397] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:20.642023 19465 raft_consensus.cc:348] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.642143 19465 raft_consensus.cc:374] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:20.642163 19465 raft_consensus.cc:729] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068a3ee55db74630a8b3f8bb0e449b42, State: Initialized, Role: FOLLOWER
I20260928 09:08:20.642272 19465 consensus_queue.cc:260] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.642517 19465 ts_tablet_manager.cc:1428] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 09:08:20.642580 19186 heartbeater.cc:503] Master 127.0.98.62:41529 was elected leader, sending a full tablet report...
W20260928 09:08:20.721899 19318 tablet.cc:2369] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:08:20.843338 19187 tablet.cc:2369] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:08:20.856858 19449 tablet.cc:2369] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:08:20.892076 19468 raft_consensus.cc:482] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:20.892274 19468 raft_consensus.cc:504] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.892833 19468 leader_election.cc:288] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f938d87f5c4132a7b3f9bf9497ee26 (127.0.98.3:42445), 068a3ee55db74630a8b3f8bb0e449b42 (127.0.98.1:42525)
I20260928 09:08:20.897112 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.893105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54449 (local address 127.0.98.1:42525)
0928 09:08:20.893149 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.893151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.893164 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:08:20.893263 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.893264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.893276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.893342 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:20.893648 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.894401 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.895866 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.896436 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.896504 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.896515 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.896994 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.897036 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:08:20.897240 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.893415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53049 (local address 127.0.98.3:42445)
0928 09:08:20.893471 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.893473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.893481 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:20.893507 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.893507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.893517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.893573 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:20.893876 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.894537 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.896146 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.896727 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.896791 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.896798 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.897108 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.897154 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:20.897382 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.893061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42525 (local address 127.0.98.2:54449)
0928 09:08:20.893114 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.893126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.893179 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.893463 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.893464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.893466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:20.893562 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.893565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.894493 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.894494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.895788 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.895791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.896857 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.896858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.896933 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.896936 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.896987 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.897030 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":29}
I20260928 09:08:20.897671 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.893120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42445 (local address 127.0.98.2:53049)
0928 09:08:20.893277 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.893289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:20.893334 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:20.893673 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:20.893674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:20.893677 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:20.893767 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.893769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.894627 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.894628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.896040 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:20.896044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.896825 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.896827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:20.896948 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.896954 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 09:08:20.897010 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 09:08:20.897058 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260928 09:08:20.897722 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aaacdddffef4ad1b8fc8e8cbd60ad35" candidate_uuid: "722635a88f3245c6948359347582d082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068a3ee55db74630a8b3f8bb0e449b42" is_pre_election: true
I20260928 09:08:20.897894 19140 raft_consensus.cc:2454] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 722635a88f3245c6948359347582d082 in term 0.
I20260928 09:08:20.898018 19402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aaacdddffef4ad1b8fc8e8cbd60ad35" candidate_uuid: "722635a88f3245c6948359347582d082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" is_pre_election: true
I20260928 09:08:20.898132 19204 leader_election.cc:302] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068a3ee55db74630a8b3f8bb0e449b42, 722635a88f3245c6948359347582d082; no voters:
I20260928 09:08:20.898167 19402 raft_consensus.cc:2454] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 722635a88f3245c6948359347582d082 in term 0.
I20260928 09:08:20.898272 19468 raft_consensus.cc:2781] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:08:20.898316 19468 raft_consensus.cc:482] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:08:20.898337 19468 raft_consensus.cc:3037] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:20.899240 19468 raft_consensus.cc:504] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.899452 19468 leader_election.cc:288] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [CANDIDATE]: Term 1 election: Requested vote from peers e9f938d87f5c4132a7b3f9bf9497ee26 (127.0.98.3:42445), 068a3ee55db74630a8b3f8bb0e449b42 (127.0.98.1:42525)
I20260928 09:08:20.899519 19402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aaacdddffef4ad1b8fc8e8cbd60ad35" candidate_uuid: "722635a88f3245c6948359347582d082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26"
I20260928 09:08:20.899587 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aaacdddffef4ad1b8fc8e8cbd60ad35" candidate_uuid: "722635a88f3245c6948359347582d082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068a3ee55db74630a8b3f8bb0e449b42"
I20260928 09:08:20.899611 19402 raft_consensus.cc:3037] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:20.899676 19140 raft_consensus.cc:3037] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:20.900578 19140 raft_consensus.cc:2454] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 722635a88f3245c6948359347582d082 in term 1.
I20260928 09:08:20.900703 19402 raft_consensus.cc:2454] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 722635a88f3245c6948359347582d082 in term 1.
I20260928 09:08:20.900781 19204 leader_election.cc:302] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068a3ee55db74630a8b3f8bb0e449b42, 722635a88f3245c6948359347582d082; no voters:
I20260928 09:08:20.900866 19468 raft_consensus.cc:2781] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:20.901006 19468 raft_consensus.cc:686] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 1 LEADER]: Becoming Leader. State: Replica: 722635a88f3245c6948359347582d082, State: Running, Role: LEADER
I20260928 09:08:20.901222 19468 consensus_queue.cc:237] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } }
I20260928 09:08:20.902185 18997 catalog_manager.cc:5495] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 reported cstate change: term changed from 0 to 1, leader changed from <none> to 722635a88f3245c6948359347582d082 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "722635a88f3245c6948359347582d082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722635a88f3245c6948359347582d082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 } health_report { overall_health: UNKNOWN } } }
I20260928 09:08:20.914259 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.911706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.0.98.2:34051)
0928 09:08:20.911774 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:20.911777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:20.911786 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:20.911825 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:20.911826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:20.911846 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:20.911934 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:20.912076 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.912571 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.913367 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:20.913652 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:20.913736 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:20.913894 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:20.913996 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:20.914104 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:20.914135 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 09:08:20.914142 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:08:20.916083 19140 raft_consensus.cc:1261] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 1 FOLLOWER]: Refusing update from remote peer 722635a88f3245c6948359347582d082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:20.916193 19402 raft_consensus.cc:1261] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 1 FOLLOWER]: Refusing update from remote peer 722635a88f3245c6948359347582d082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:20.916491 19472 consensus_queue.cc:1035] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068a3ee55db74630a8b3f8bb0e449b42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:20.916647 19468 consensus_queue.cc:1035] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f938d87f5c4132a7b3f9bf9497ee26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:20.919531 19477 mvcc.cc:205] Tried to move back new op lower bound from 7334242307749163008 to 7334242307691986944. Current Snapshot: MvccSnapshot[applied={T|T < 7334242307749163008}]
I20260928 09:08:20.920071 19476 mvcc.cc:205] Tried to move back new op lower bound from 7334242307749163008 to 7334242307691986944. Current Snapshot: MvccSnapshot[applied={T|T < 7334242307749163008}]
I20260928 09:08:20.921090 18997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:20.921187 18997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:20.922003 18997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4877badc20ba49188d4f846f4f2f4e73: Sending DeleteTablet for 3 replicas of tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35
I20260928 09:08:20.922291 19251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:20 UTC) from {username='slave'} at 127.0.0.1:34286
I20260928 09:08:20.922338 19120 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:20 UTC) from {username='slave'} at 127.0.0.1:49664
I20260928 09:08:20.922466 19382 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:20 UTC) from {username='slave'} at 127.0.0.1:50730
I20260928 09:08:20.922567 19486 tablet_replica.cc:330] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: stopping tablet replica
I20260928 09:08:20.922658 19486 raft_consensus.cc:2229] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:20.922675 19487 tablet_replica.cc:330] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: stopping tablet replica
I20260928 09:08:20.922758 19487 raft_consensus.cc:2229] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:20.922777 19486 raft_consensus.cc:2258] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:20.922854 19487 raft_consensus.cc:2258] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:20.923214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19058
I20260928 09:08:20.923260 19486 ts_tablet_manager.cc:1905] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 068a3ee55db74630a8b3f8bb0e449b42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:20.923286 19487 ts_tablet_manager.cc:1905] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:20.925248 19487 ts_tablet_manager.cc:1918] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:20.925313 19487 log.cc:1196] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/2aaacdddffef4ad1b8fc8e8cbd60ad35
I20260928 09:08:20.925589 19487 ts_tablet_manager.cc:1939] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P e9f938d87f5c4132a7b3f9bf9497ee26: Deleting consensus metadata
I20260928 09:08:20.926812 18985 catalog_manager.cc:4841] TS e9f938d87f5c4132a7b3f9bf9497ee26 (127.0.98.3:42445): tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35 (table test-table [id=d09b169d2a4f4c139788f917b61a6ead]) successfully deleted
I20260928 09:08:20.929713 19485 tablet_replica.cc:330] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: stopping tablet replica
I20260928 09:08:20.929845 19485 raft_consensus.cc:2229] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:08:20.929951 19485 raft_consensus.cc:2258] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:20.930320 19485 ts_tablet_manager.cc:1905] T 2aaacdddffef4ad1b8fc8e8cbd60ad35 P 722635a88f3245c6948359347582d082: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 09:08:20.930795 18982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42525 (error 108)
I20260928 09:08:20.930919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19189
I20260928 09:08:20.932492 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.932326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42525 (local address 127.0.0.1:49668)
0928 09:08:20.932397 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.932450 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 09:08:20.932646 18982 catalog_manager.cc:4584] TS 068a3ee55db74630a8b3f8bb0e449b42 (127.0.98.1:42525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2aaacdddffef4ad1b8fc8e8cbd60ad35: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42525: connect: Connection refused (error 111)
I20260928 09:08:20.936012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19320
I20260928 09:08:20.936044 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:20.935928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34051 (local address 127.0.0.1:34294)
0928 09:08:20.935984 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:20.936019 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:08:20.940781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 18967
2026-09-28T09:08:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (940 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:21 dist-test-slave-smns krb5kdc[19491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:21 dist-test-slave-smns krb5kdc[19491](info): set up 2 sockets
Sep 28 09:08:21 dist-test-slave-smns krb5kdc[19491](info): commencing operation
krb5kdc: starting...
W20260928 09:08:21.981623 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:22.104766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46505
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:22.185895 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:22.186182 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:22.186230 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:22.187479 19504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:22.187580 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:22.187608 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:22.187637 19504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:22.187690 19504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:22.189227 19504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:22.189507 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:22.189764 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:08:22.194854 19504 server_base.cc:1034] running on GCE node
W20260928 09:08:22.194844 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.194834 19512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.194826 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:22.195462 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:22.195735 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:22.196907 19504 hybrid_clock.cc:648] HybridClock initialized: now 1790586502196893 us; error 44 us; skew 500 ppm
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:22.199646 19504 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:22.200111 19504 webserver.cc:460] Webserver started at http://127.0.98.62:34695/ using document root <none> and password file <none>
I20260928 09:08:22.200299 19504 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:22.200341 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:22.200431 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:22.201467 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "3581afcc4d324865a5c12fab862bf0ae"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "04d0d966a23880f905a3e198ea481c3d"
server_key_iv: "a393ffc302ea5c3422b12fbb60449b27"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.201846 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "3581afcc4d324865a5c12fab862bf0ae"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "04d0d966a23880f905a3e198ea481c3d"
server_key_iv: "a393ffc302ea5c3422b12fbb60449b27"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.203202 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:22.203963 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.204136 19504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:22.204221 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "3581afcc4d324865a5c12fab862bf0ae"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "04d0d966a23880f905a3e198ea481c3d"
server_key_iv: "a393ffc302ea5c3422b12fbb60449b27"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.204285 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:22.217114 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:22.219353 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:22.219463 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:22.223295 19504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46505
I20260928 09:08:22.223330 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46505 every 8 connection(s)
I20260928 09:08:22.223729 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:22.224697 19572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:22.227151 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae: Bootstrap starting.
I20260928 09:08:22.227752 19572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:22.228044 19572 log.cc:824] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:22.228768 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae: No bootstrap required, opened a new log
I20260928 09:08:22.229478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19504
I20260928 09:08:22.229638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:22.229857 392 external_mini_cluster.cc:1428] Setting key 2efaf34c8812aad32f89cbb2c0623617
I20260928 09:08:22.231041 19572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3581afcc4d324865a5c12fab862bf0ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46505 } }
I20260928 09:08:22.231207 19572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:22.231245 19572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3581afcc4d324865a5c12fab862bf0ae, State: Initialized, Role: FOLLOWER
I20260928 09:08:22.231419 19572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3581afcc4d324865a5c12fab862bf0ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46505 } }
I20260928 09:08:22.231521 19572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:22.231561 19572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:22.231609 19572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:22.232649 19572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3581afcc4d324865a5c12fab862bf0ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46505 } }
I20260928 09:08:22.232750 19572 leader_election.cc:302] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3581afcc4d324865a5c12fab862bf0ae; no voters:
I20260928 09:08:22.232957 19572 leader_election.cc:288] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:22.232977 19577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:22.233090 19577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [term 1 LEADER]: Becoming Leader. State: Replica: 3581afcc4d324865a5c12fab862bf0ae, State: Running, Role: LEADER
I20260928 09:08:22.233235 19572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:22.233250 19577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3581afcc4d324865a5c12fab862bf0ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46505 } }
I20260928 09:08:22.233836 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3581afcc4d324865a5c12fab862bf0ae. Latest consensus state: current_term: 1 leader_uuid: "3581afcc4d324865a5c12fab862bf0ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3581afcc4d324865a5c12fab862bf0ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46505 } } }
I20260928 09:08:22.233841 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3581afcc4d324865a5c12fab862bf0ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3581afcc4d324865a5c12fab862bf0ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46505 } } }
I20260928 09:08:22.233964 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:22.233968 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:22.234316 19586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:22.234905 19586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:22.236765 19586 catalog_manager.cc:1347] Generated new cluster ID: 9eb2232b4aa346a694e66ae64f3e3da4
I20260928 09:08:22.236821 19586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:22.241642 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.230416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.98.62:46505)
0928 09:08:22.230592 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.230601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.230633 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:08:22.231021 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.231028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.231080 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.231235 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:22.232046 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.233166 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.234301 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.234910 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.235389 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.237858 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:22.237861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:22.237862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:22.237874 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:22.239598 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.239991 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.239994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.239995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.240107 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.240323 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.240323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.240324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.240569 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:22.241249 (+ 680us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.241310 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.241424 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260928 09:08:22.248894 19586 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:22.249624 19586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:22.260908 19586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae: Generated new TSK 0
I20260928 09:08:22.261124 19586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:08:22.297641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46505
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:22.379264 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:22.379505 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:22.379542 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:22.380699 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:22.380766 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:22.380810 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:08:22.382380 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:22.382642 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:22.382867 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:22.388114 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.388142 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:22.388232 19598 server_base.cc:1034] running on GCE node
W20260928 09:08:22.388114 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:22.388523 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:22.388765 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:22.389935 19598 hybrid_clock.cc:648] HybridClock initialized: now 1790586502389928 us; error 37 us; skew 500 ppm
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:22.392484 19598 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:08:22.392930 19598 webserver.cc:460] Webserver started at http://127.0.98.1:45541/ using document root <none> and password file <none>
I20260928 09:08:22.393153 19598 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:22.393236 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:22.393391 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:22.394515 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "c95662d5886240db9a6c38385b0976e6"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "6c75a3921a41c6c9340343db57cb109b"
server_key_iv: "40a64360eb4ba15088810ec299457759"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.394919 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c95662d5886240db9a6c38385b0976e6"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "6c75a3921a41c6c9340343db57cb109b"
server_key_iv: "40a64360eb4ba15088810ec299457759"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.396251 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:22.396934 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.397115 19598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:22.397220 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "c95662d5886240db9a6c38385b0976e6"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "6c75a3921a41c6c9340343db57cb109b"
server_key_iv: "40a64360eb4ba15088810ec299457759"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.397311 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:22.403517 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:22.405269 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:22.405426 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:22.405689 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:22.406021 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:22.406075 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.406116 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:22.406142 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.411898 19598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45965
I20260928 09:08:22.411969 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45965 every 8 connection(s)
I20260928 09:08:22.412290 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:08:22.412570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19598
I20260928 09:08:22.412683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:08:22.412909 392 external_mini_cluster.cc:1428] Setting key 465f89b8306bece31e2969f17de13ab1
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 09:08:22.423398 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.413088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52421 (local address 127.0.98.62:46505)
0928 09:08:22.413132 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.413134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.413453 (+ 319us) server_negotiation.cc:407] Connection header received
0928 09:08:22.414936 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.414936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.414949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.415003 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:22.415715 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.416367 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.417339 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.417654 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.417750 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.420181 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:22.420182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:22.420183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:22.420190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:22.421896 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.422368 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.422370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.422370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.422435 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.422717 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.422718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.422719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.422843 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:22.422919 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.423198 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.423294 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:08:22.423599 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.413088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46505 (local address 127.0.98.1:52421)
0928 09:08:22.413331 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.413367 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.414822 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.415082 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.415087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.415425 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:22.415612 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.415616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.416448 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.416449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.417243 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.417244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.417754 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.417755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.417894 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.418459 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:22.418467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:22.419973 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:22.422004 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:22.422011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:22.422013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:22.422270 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:22.422502 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:22.422504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:22.422504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:22.422643 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:22.422910 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:22.422916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:22.423061 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.423176 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.423342 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 09:08:22.424122 19727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46505
I20260928 09:08:22.424221 19727 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:22.424437 19727 heartbeater.cc:511] Master 127.0.98.62:46505 requested a full tablet report, sending...
I20260928 09:08:22.424980 19536 ts_manager.cc:194] Registered new tserver with Master: c95662d5886240db9a6c38385b0976e6 (127.0.98.1:45965)
I20260928 09:08:22.425660 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:08:22.440433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46505
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:22.521813 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:22.522017 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:22.522042 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:22.523149 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:22.523188 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:22.523214 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:22.524641 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:22.524842 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:22.525015 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:08:22.530193 19732 server_base.cc:1034] running on GCE node
W20260928 09:08:22.530238 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.530135 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.530128 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:22.530671 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:22.530917 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:22.532063 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790586502532050 us; error 31 us; skew 500 ppm
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:22.534931 19732 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:08:22.535499 19732 webserver.cc:460] Webserver started at http://127.0.98.2:36207/ using document root <none> and password file <none>
I20260928 09:08:22.535792 19732 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:22.535872 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:22.536010 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:22.537107 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "d89d69c393fc42458dcf142822fa6f29"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "d3c615f4a59c050a2afd2d47cd548557"
server_key_iv: "713ecdd6c11c62a183e83829cfb04e6e"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.537493 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d89d69c393fc42458dcf142822fa6f29"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "d3c615f4a59c050a2afd2d47cd548557"
server_key_iv: "713ecdd6c11c62a183e83829cfb04e6e"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.538879 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:22.539644 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.539994 19732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:22.540112 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "d89d69c393fc42458dcf142822fa6f29"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "d3c615f4a59c050a2afd2d47cd548557"
server_key_iv: "713ecdd6c11c62a183e83829cfb04e6e"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.540201 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:22.555552 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:22.557921 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:22.558099 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:22.558346 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:22.558682 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:22.558748 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.558789 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:22.558816 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.564721 19732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41611
I20260928 09:08:22.564815 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41611 every 8 connection(s)
I20260928 09:08:22.565200 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:22.574493 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.566004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60773 (local address 127.0.98.62:46505)
0928 09:08:22.566070 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.566072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.566250 (+ 178us) server_negotiation.cc:407] Connection header received
0928 09:08:22.567218 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.567220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.567231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.567296 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:22.567937 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.568464 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.569450 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.569655 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.569709 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.571787 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:22.571789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:22.571789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:22.571796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:22.573223 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.573644 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.573645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.573646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.573697 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.573942 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.573943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.573944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.574057 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:22.574116 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.574357 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.574427 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:08:22.574685 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.565967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46505 (local address 127.0.98.2:60773)
0928 09:08:22.566118 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.566159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.567079 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.567372 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.567376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.567635 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:22.567853 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.567857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.568547 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.568548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.569374 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.569376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.569726 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.569727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.569847 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.570292 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:22.570299 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:22.571690 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:22.573302 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:22.573308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:22.573310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:22.573557 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:22.573758 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:22.573758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:22.573759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:22.573866 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:22.574115 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:22.574119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:22.574235 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.574326 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.574448 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260928 09:08:22.575273 19861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46505
I20260928 09:08:22.575384 19861 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:22.575410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19732
I20260928 09:08:22.575517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:08:22.575592 19861 heartbeater.cc:511] Master 127.0.98.62:46505 requested a full tablet report, sending...
I20260928 09:08:22.575717 392 external_mini_cluster.cc:1428] Setting key f9ec3fde8fb62f2000d7076de77eaf7d
I20260928 09:08:22.576063 19536 ts_manager.cc:194] Registered new tserver with Master: d89d69c393fc42458dcf142822fa6f29 (127.0.98.2:41611)
I20260928 09:08:22.576632 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60773
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:08:22.602277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46505
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:22.685490 19866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:22.685806 19866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:22.685848 19866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:22.687182 19866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:22.687253 19866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:22.687309 19866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:22.688895 19866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:22.689172 19866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:22.689399 19866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:22.694763 19872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.694766 19871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:22.694794 19874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:22.695025 19866 server_base.cc:1034] running on GCE node
I20260928 09:08:22.695272 19866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:22.695552 19866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:22.696753 19866 hybrid_clock.cc:648] HybridClock initialized: now 1790586502696755 us; error 46 us; skew 500 ppm
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:22.699419 19866 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:08:22.699889 19866 webserver.cc:460] Webserver started at http://127.0.98.3:40087/ using document root <none> and password file <none>
I20260928 09:08:22.700089 19866 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:22.700157 19866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:22.700320 19866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:22.701418 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "a126aa5319eb4225a453248fa099fd1d"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "d1f1a727e35c58db01ad2fd3767adba5"
server_key_iv: "3b4e22a69b958213940bb28736d35083"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.701834 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a126aa5319eb4225a453248fa099fd1d"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "d1f1a727e35c58db01ad2fd3767adba5"
server_key_iv: "3b4e22a69b958213940bb28736d35083"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.703198 19866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:22.703931 19881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.704118 19866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:22.704216 19866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "a126aa5319eb4225a453248fa099fd1d"
format_stamp: "Formatted at 2026-09-28 09:08:22 on dist-test-slave-smns"
server_key: "d1f1a727e35c58db01ad2fd3767adba5"
server_key_iv: "3b4e22a69b958213940bb28736d35083"
server_key_version: "encryptionkey@0"
I20260928 09:08:22.704290 19866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:22.719746 19866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:22.721698 19866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:22.721855 19866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:22.722090 19866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:22.722404 19866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:22.722440 19866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.722462 19866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:22.722476 19866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:22.728314 19866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43159
I20260928 09:08:22.728372 19994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43159 every 8 connection(s)
I20260928 09:08:22.728675 19866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:08:22 dist-test-slave-smns krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:22.737089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 19866
I20260928 09:08:22.737239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:08:22.737457 392 external_mini_cluster.cc:1428] Setting key fbdb8d0dc97672f12b8705f95c50f18f
I20260928 09:08:22.738708 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.729367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43575 (local address 127.0.98.62:46505)
0928 09:08:22.729433 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.729435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.729676 (+ 241us) server_negotiation.cc:407] Connection header received
0928 09:08:22.730632 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.730633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.730644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.730705 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:22.731352 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.731849 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.732945 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.733142 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.733198 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.735545 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:22.735547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:22.735547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:22.735553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:22.737029 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.737614 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.737615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.737616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.737674 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.738038 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.738039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.738040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.738185 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:22.738254 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.738578 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.738644 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:08:22.738966 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.729356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46505 (local address 127.0.98.3:43575)
0928 09:08:22.729528 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.729585 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.730521 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.730778 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.730781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.731037 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:22.731275 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.731280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.731959 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.731961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.732793 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.732796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.733309 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.733310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.733425 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.734012 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:22.734024 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:22.735458 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:22.737119 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:22.737125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:22.737127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:22.737473 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:22.737792 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:22.737793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:22.737794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:22.737969 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:22.738259 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:22.738265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:22.738435 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.738551 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.738691 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260928 09:08:22.739480 19995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46505
I20260928 09:08:22.739612 19995 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:22.739773 19995 heartbeater.cc:511] Master 127.0.98.62:46505 requested a full tablet report, sending...
I20260928 09:08:22.740113 19536 ts_manager.cc:194] Registered new tserver with Master: a126aa5319eb4225a453248fa099fd1d (127.0.98.3:43159)
I20260928 09:08:22.740547 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43575
I20260928 09:08:22.749611 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:08:22.757833 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.751096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.0.98.62:46505)
0928 09:08:22.751185 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.751187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.751296 (+ 109us) server_negotiation.cc:407] Connection header received
0928 09:08:22.751396 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.751397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.751407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.751453 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:22.752024 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.752527 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.753315 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.753527 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.753629 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.754861 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:22.754863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:22.754863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:22.754871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:22.756154 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.756524 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.756526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.756526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.756580 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:22.756801 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:22.756802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:22.756802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:22.757001 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:22.757095 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.757431 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.757602 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:08:22.759555 19536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:08:22.761582 19536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:08:22.768106 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.764046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41611 (local address 127.0.0.1:39398)
0928 09:08:22.764360 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.764370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.764408 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.764683 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.764686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.764688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:22.764782 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.764783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.765735 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.765736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.766940 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.766943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.767787 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.767788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.767865 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.767880 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.767953 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.768000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":60,"threads_started":1}
I20260928 09:08:22.768132 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.764188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.98.2:41611)
0928 09:08:22.764379 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.764386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.764399 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:08:22.764478 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.764481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.764498 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.764575 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:22.764864 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.765622 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.767031 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.767697 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.767775 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.767837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.767979 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.768024 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260928 09:08:22.768333 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.763832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43159 (local address 127.0.0.1:39552)
0928 09:08:22.764139 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.764185 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.764249 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.764696 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.764697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.764698 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:22.764762 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.764765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.766160 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.766161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.767210 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.767212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.768117 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.768118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.768187 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.768192 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.768225 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.768265 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260928 09:08:22.768661 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.764018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.0.98.3:43159)
0928 09:08:22.764537 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.764541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.764550 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:22.764585 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.764588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.764604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.764843 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:22.764933 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.766037 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.767287 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.768018 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.768453 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.768525 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.768568 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.768602 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":72,"threads_started":1}
I20260928 09:08:22.768903 19795 tablet_service.cc:1467] Processing CreateTablet for tablet b05316ee0d7648cdacd355ad0b5bf5a9 (DEFAULT_TABLE table=test-table [id=f35efe7bb77749a3866562c705f50091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:22.769275 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05316ee0d7648cdacd355ad0b5bf5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:22.769358 19929 tablet_service.cc:1467] Processing CreateTablet for tablet b05316ee0d7648cdacd355ad0b5bf5a9 (DEFAULT_TABLE table=test-table [id=f35efe7bb77749a3866562c705f50091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:22.769727 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05316ee0d7648cdacd355ad0b5bf5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:22.769850 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.763832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45965 (local address 127.0.0.1:43572)
0928 09:08:22.764152 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.764186 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.764251 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.764758 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.764759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.764764 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:22.764842 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.764844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.767280 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.767280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.768034 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.768036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.769599 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.769601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.769689 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.769693 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.769739 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.769782 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260928 09:08:22.770195 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.763943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.0.98.1:45965)
0928 09:08:22.764439 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.764444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.764517 (+ 73us) server_negotiation.cc:407] Connection header received
0928 09:08:22.764563 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.764566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.764580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.764655 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:22.764933 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.766038 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.768125 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.769477 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.769985 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.770053 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.770079 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.770120 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":90,"threads_started":1}
I20260928 09:08:22.770838 19661 tablet_service.cc:1467] Processing CreateTablet for tablet b05316ee0d7648cdacd355ad0b5bf5a9 (DEFAULT_TABLE table=test-table [id=f35efe7bb77749a3866562c705f50091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:22.771163 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05316ee0d7648cdacd355ad0b5bf5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:22.772506 20012 tablet_bootstrap.cc:492] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Bootstrap starting.
I20260928 09:08:22.772949 20013 tablet_bootstrap.cc:492] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Bootstrap starting.
I20260928 09:08:22.773290 20012 tablet_bootstrap.cc:654] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:22.773674 20013 tablet_bootstrap.cc:654] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:22.773924 20012 log.cc:824] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:22.774317 20013 log.cc:824] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:22.774855 20012 tablet_bootstrap.cc:492] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: No bootstrap required, opened a new log
I20260928 09:08:22.774927 20012 ts_tablet_manager.cc:1397] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:08:22.775004 20013 tablet_bootstrap.cc:492] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: No bootstrap required, opened a new log
I20260928 09:08:22.775068 20013 ts_tablet_manager.cc:1397] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:08:22.776932 20012 raft_consensus.cc:348] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.776932 20013 raft_consensus.cc:348] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.777119 20012 raft_consensus.cc:374] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:22.777119 20013 raft_consensus.cc:374] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:22.777158 20012 raft_consensus.cc:729] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89d69c393fc42458dcf142822fa6f29, State: Initialized, Role: FOLLOWER
I20260928 09:08:22.777158 20013 raft_consensus.cc:729] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a126aa5319eb4225a453248fa099fd1d, State: Initialized, Role: FOLLOWER
I20260928 09:08:22.777283 20012 consensus_queue.cc:260] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.777283 20013 consensus_queue.cc:260] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.777587 20013 ts_tablet_manager.cc:1428] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 09:08:22.777592 20012 ts_tablet_manager.cc:1428] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:22.777650 19861 heartbeater.cc:503] Master 127.0.98.62:46505 was elected leader, sending a full tablet report...
I20260928 09:08:22.777722 19995 heartbeater.cc:503] Master 127.0.98.62:46505 was elected leader, sending a full tablet report...
I20260928 09:08:22.777745 20015 tablet_bootstrap.cc:492] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: Bootstrap starting.
I20260928 09:08:22.778414 20015 tablet_bootstrap.cc:654] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:22.779026 20015 log.cc:824] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:22.779783 20015 tablet_bootstrap.cc:492] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: No bootstrap required, opened a new log
I20260928 09:08:22.779838 20015 ts_tablet_manager.cc:1397] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 09:08:22.781152 20015 raft_consensus.cc:348] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.781287 20015 raft_consensus.cc:374] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:22.781311 20015 raft_consensus.cc:729] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95662d5886240db9a6c38385b0976e6, State: Initialized, Role: FOLLOWER
I20260928 09:08:22.781422 20015 consensus_queue.cc:260] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.781705 20015 ts_tablet_manager.cc:1428] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:08:22.781770 19727 heartbeater.cc:503] Master 127.0.98.62:46505 was elected leader, sending a full tablet report...
W20260928 09:08:22.815840 19862 tablet.cc:2369] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:08:22.913208 19728 tablet.cc:2369] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:08:22.979274 19996 tablet.cc:2369] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:08:22.981160 20020 raft_consensus.cc:482] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:22.981282 20020 raft_consensus.cc:504] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.981660 20020 leader_election.cc:288] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a126aa5319eb4225a453248fa099fd1d (127.0.98.3:43159), d89d69c393fc42458dcf142822fa6f29 (127.0.98.2:41611)
I20260928 09:08:22.984892 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.982193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36889 (local address 127.0.98.2:41611)
0928 09:08:22.982265 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.982267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.982294 (+ 27us) server_negotiation.cc:407] Connection header received
0928 09:08:22.982369 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.982370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.982382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.982436 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:22.982604 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.983092 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.984147 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.984526 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.984611 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.984622 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.984786 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.984823 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:08:22.984911 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.982095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41611 (local address 127.0.98.1:36889)
0928 09:08:22.982187 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.982197 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.982314 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.982442 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.982443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.982445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:22.982533 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.982535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.983179 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.983180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.984042 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.984045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.984611 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.984612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.984684 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.984688 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.984777 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.984820 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":32,"thread_start_us":57,"threads_started":1}
I20260928 09:08:22.985011 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.981970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50165 (local address 127.0.98.3:43159)
0928 09:08:22.982069 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:22.982072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:22.982254 (+ 182us) server_negotiation.cc:407] Connection header received
0928 09:08:22.982362 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:22.982364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:22.982374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:22.982454 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:22.982634 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.983097 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.984173 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.984546 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.984639 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.984649 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:22.984904 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 09:08:22.984945 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 09:08:22.985028 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:22.981883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43159 (local address 127.0.98.1:50165)
0928 09:08:22.982074 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:22.982086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:22.982277 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:22.982457 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:22.982458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:22.982459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:22.982554 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.982556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.983182 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.983183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.984060 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:22.984062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:22.984754 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:22.984754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:22.984812 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:22.984814 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:08:22.984894 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:08:22.984934 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":32,"thread_start_us":85,"threads_started":1}
I20260928 09:08:22.985610 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05316ee0d7648cdacd355ad0b5bf5a9" candidate_uuid: "c95662d5886240db9a6c38385b0976e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a126aa5319eb4225a453248fa099fd1d" is_pre_election: true
I20260928 09:08:22.985606 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05316ee0d7648cdacd355ad0b5bf5a9" candidate_uuid: "c95662d5886240db9a6c38385b0976e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89d69c393fc42458dcf142822fa6f29" is_pre_election: true
I20260928 09:08:22.985781 19949 raft_consensus.cc:2454] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95662d5886240db9a6c38385b0976e6 in term 0.
I20260928 09:08:22.985781 19815 raft_consensus.cc:2454] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95662d5886240db9a6c38385b0976e6 in term 0.
I20260928 09:08:22.986001 19616 leader_election.cc:302] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a126aa5319eb4225a453248fa099fd1d, c95662d5886240db9a6c38385b0976e6; no voters:
I20260928 09:08:22.986135 20020 raft_consensus.cc:2781] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:08:22.986191 20020 raft_consensus.cc:482] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:08:22.986214 20020 raft_consensus.cc:3037] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:22.986896 20020 raft_consensus.cc:504] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.987035 20020 leader_election.cc:288] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [CANDIDATE]: Term 1 election: Requested vote from peers a126aa5319eb4225a453248fa099fd1d (127.0.98.3:43159), d89d69c393fc42458dcf142822fa6f29 (127.0.98.2:41611)
I20260928 09:08:22.987215 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05316ee0d7648cdacd355ad0b5bf5a9" candidate_uuid: "c95662d5886240db9a6c38385b0976e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a126aa5319eb4225a453248fa099fd1d"
I20260928 09:08:22.987243 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05316ee0d7648cdacd355ad0b5bf5a9" candidate_uuid: "c95662d5886240db9a6c38385b0976e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89d69c393fc42458dcf142822fa6f29"
I20260928 09:08:22.987303 19949 raft_consensus.cc:3037] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:22.987310 19815 raft_consensus.cc:3037] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:22.988201 19815 raft_consensus.cc:2454] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95662d5886240db9a6c38385b0976e6 in term 1.
I20260928 09:08:22.988201 19949 raft_consensus.cc:2454] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95662d5886240db9a6c38385b0976e6 in term 1.
I20260928 09:08:22.988399 19616 leader_election.cc:302] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a126aa5319eb4225a453248fa099fd1d, c95662d5886240db9a6c38385b0976e6; no voters:
I20260928 09:08:22.988485 20020 raft_consensus.cc:2781] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:22.988662 20020 raft_consensus.cc:686] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [term 1 LEADER]: Becoming Leader. State: Replica: c95662d5886240db9a6c38385b0976e6, State: Running, Role: LEADER
I20260928 09:08:22.988763 20020 consensus_queue.cc:237] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } }
I20260928 09:08:22.989619 19536 catalog_manager.cc:5495] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c95662d5886240db9a6c38385b0976e6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c95662d5886240db9a6c38385b0976e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95662d5886240db9a6c38385b0976e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 } health_report { overall_health: UNKNOWN } } }
I20260928 09:08:23.050082 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.047427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.0.98.1:45965)
0928 09:08:23.047494 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.047496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.047504 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:23.047581 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.047582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.047603 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.047665 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:23.047836 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.048381 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.049168 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.049355 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.049461 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.049734 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:23.049851 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:23.049919 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.049950 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.049970 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:08:23.052604 19949 raft_consensus.cc:1261] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 1 FOLLOWER]: Refusing update from remote peer c95662d5886240db9a6c38385b0976e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:23.052574 19815 raft_consensus.cc:1261] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 1 FOLLOWER]: Refusing update from remote peer c95662d5886240db9a6c38385b0976e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:23.052974 20020 consensus_queue.cc:1035] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d89d69c393fc42458dcf142822fa6f29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:23.053215 20023 consensus_queue.cc:1035] T b05316ee0d7648cdacd355ad0b5bf5a9 P c95662d5886240db9a6c38385b0976e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a126aa5319eb4225a453248fa099fd1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:23.056241 20028 mvcc.cc:205] Tried to move back new op lower bound from 7334242316499283968 to 7334242316242575360. Current Snapshot: MvccSnapshot[applied={T|T < 7334242316499283968}]
I20260928 09:08:23.056522 20027 mvcc.cc:205] Tried to move back new op lower bound from 7334242316499283968 to 7334242316242575360. Current Snapshot: MvccSnapshot[applied={T|T < 7334242316499283968}]
I20260928 09:08:23.058487 19535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:23.058590 19535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:23.059417 19535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3581afcc4d324865a5c12fab862bf0ae: Sending DeleteTablet for 3 replicas of tablet b05316ee0d7648cdacd355ad0b5bf5a9
I20260928 09:08:23.059736 19661 tablet_service.cc:1514] Processing DeleteTablet for tablet b05316ee0d7648cdacd355ad0b5bf5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43572
I20260928 09:08:23.059772 19929 tablet_service.cc:1514] Processing DeleteTablet for tablet b05316ee0d7648cdacd355ad0b5bf5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39552
I20260928 09:08:23.059846 19795 tablet_service.cc:1514] Processing DeleteTablet for tablet b05316ee0d7648cdacd355ad0b5bf5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39398
I20260928 09:08:23.060060 20037 tablet_replica.cc:330] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: stopping tablet replica
I20260928 09:08:23.060073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19598
I20260928 09:08:23.060616 20037 raft_consensus.cc:2229] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:23.060773 20037 raft_consensus.cc:2258] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:23.061229 20037 ts_tablet_manager.cc:1905] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:23.062091 20036 tablet_replica.cc:330] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: stopping tablet replica
I20260928 09:08:23.062208 20036 raft_consensus.cc:2229] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:23.062338 20036 raft_consensus.cc:2258] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:23.062755 20036 ts_tablet_manager.cc:1905] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:23.063413 20037 ts_tablet_manager.cc:1918] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:23.063494 20037 log.cc:1196] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/b05316ee0d7648cdacd355ad0b5bf5a9
I20260928 09:08:23.063766 20037 ts_tablet_manager.cc:1939] T b05316ee0d7648cdacd355ad0b5bf5a9 P d89d69c393fc42458dcf142822fa6f29: Deleting consensus metadata
I20260928 09:08:23.064549 19522 catalog_manager.cc:4841] TS d89d69c393fc42458dcf142822fa6f29 (127.0.98.2:41611): tablet b05316ee0d7648cdacd355ad0b5bf5a9 (table test-table [id=f35efe7bb77749a3866562c705f50091]) successfully deleted
I20260928 09:08:23.064605 20036 ts_tablet_manager.cc:1918] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:23.064689 20036 log.cc:1196] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/b05316ee0d7648cdacd355ad0b5bf5a9
I20260928 09:08:23.064945 20036 ts_tablet_manager.cc:1939] T b05316ee0d7648cdacd355ad0b5bf5a9 P a126aa5319eb4225a453248fa099fd1d: Deleting consensus metadata
I20260928 09:08:23.065346 19522 catalog_manager.cc:4841] TS a126aa5319eb4225a453248fa099fd1d (127.0.98.3:43159): tablet b05316ee0d7648cdacd355ad0b5bf5a9 (table test-table [id=f35efe7bb77749a3866562c705f50091]) successfully deleted
W20260928 09:08:23.066036 19520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45965 (error 108)
I20260928 09:08:23.066294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19732
I20260928 09:08:23.066610 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.066489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45965 (local address 127.0.0.1:43584)
0928 09:08:23.066537 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.066572 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 09:08:23.066735 19520 catalog_manager.cc:4584] TS c95662d5886240db9a6c38385b0976e6 (127.0.98.1:45965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b05316ee0d7648cdacd355ad0b5bf5a9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45965: connect: Connection refused (error 111)
I20260928 09:08:23.071965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19866
I20260928 09:08:23.076488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 19504
2026-09-28T09:08:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2136 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T09:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:23Z Disabled control of system clock
I20260928 09:08:23.104164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42793
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:23.186851 20042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:23.187137 20042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:23.187188 20042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:23.188405 20042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:23.188498 20042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:23.188521 20042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:23.188542 20042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:23.188602 20042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:23.190043 20042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:23.190374 20042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:23.190682 20042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:23.195715 20050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.195776 20048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:23.195819 20042 server_base.cc:1034] running on GCE node
W20260928 09:08:23.195716 20047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:23.196213 20042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:23.196555 20042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:23.197713 20042 hybrid_clock.cc:648] HybridClock initialized: now 1790586503197697 us; error 49 us; skew 500 ppm
I20260928 09:08:23.199015 20042 webserver.cc:460] Webserver started at http://127.0.98.62:34063/ using document root <none> and password file <none>
I20260928 09:08:23.199306 20042 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:23.199379 20042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:23.199529 20042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:23.200608 20042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "f75c7df09822486baa93d2581d81b3df"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "fa2efbedddd0091c8ce91701dab7daf3"
server_key_iv: "f33ba815a6b97a89a21ee45d17a18a5d"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.200978 20042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "f75c7df09822486baa93d2581d81b3df"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "fa2efbedddd0091c8ce91701dab7daf3"
server_key_iv: "f33ba815a6b97a89a21ee45d17a18a5d"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.202389 20042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:23.203092 20056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.203274 20042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:23.203379 20042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "f75c7df09822486baa93d2581d81b3df"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "fa2efbedddd0091c8ce91701dab7daf3"
server_key_iv: "f33ba815a6b97a89a21ee45d17a18a5d"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.203455 20042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:23.212198 20042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:23.212504 20042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:23.212651 20042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:23.216841 20042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42793
I20260928 09:08:23.216871 20108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42793 every 8 connection(s)
I20260928 09:08:23.217250 20042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:23.218255 20109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:23.218284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20042
I20260928 09:08:23.218430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:23.218633 392 external_mini_cluster.cc:1428] Setting key d004d1c7f7fa2336a6c33d2bf09df0d9
I20260928 09:08:23.221796 20109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df: Bootstrap starting.
I20260928 09:08:23.222397 20109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:23.222698 20109 log.cc:824] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:23.223695 20109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df: No bootstrap required, opened a new log
I20260928 09:08:23.224823 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.219156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.98.62:42793)
0928 09:08:23.219352 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.219362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.219394 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:08:23.219897 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.219907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.220004 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.220171 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:23.220393 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.221801 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.222641 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.223258 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.223400 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.224162 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:23.224166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:23.224166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:23.224346 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:23.224381 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:23.224392 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:23.224465 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.224566 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.224648 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260928 09:08:23.225996 20109 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75c7df09822486baa93d2581d81b3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260928 09:08:23.226225 20109 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:23.226253 20109 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75c7df09822486baa93d2581d81b3df, State: Initialized, Role: FOLLOWER
I20260928 09:08:23.226390 20109 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75c7df09822486baa93d2581d81b3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260928 09:08:23.226468 20109 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:23.226512 20109 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:23.226564 20109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:23.227157 20109 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75c7df09822486baa93d2581d81b3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260928 09:08:23.227248 20109 leader_election.cc:302] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75c7df09822486baa93d2581d81b3df; no voters:
I20260928 09:08:23.227445 20109 leader_election.cc:288] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:23.227527 20114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:23.227671 20114 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [term 1 LEADER]: Becoming Leader. State: Replica: f75c7df09822486baa93d2581d81b3df, State: Running, Role: LEADER
I20260928 09:08:23.227738 20109 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:23.227779 20114 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75c7df09822486baa93d2581d81b3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260928 09:08:23.228164 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75c7df09822486baa93d2581d81b3df. Latest consensus state: current_term: 1 leader_uuid: "f75c7df09822486baa93d2581d81b3df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75c7df09822486baa93d2581d81b3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } }
I20260928 09:08:23.228159 20115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75c7df09822486baa93d2581d81b3df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75c7df09822486baa93d2581d81b3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } }
I20260928 09:08:23.228273 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:23.228276 20115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:23.228571 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:23.229113 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:23.230794 20121 catalog_manager.cc:1347] Generated new cluster ID: 91dc8e6101734412ab8e3e33d5cab50a
I20260928 09:08:23.230895 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:23.239974 20121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:23.240643 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:23.243486 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df: Generated new TSK 0
I20260928 09:08:23.243680 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:23.250468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42793
--builtin_ntp_servers=127.0.98.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:23.333328 20133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:23.333596 20133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:23.333665 20133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:23.334841 20133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:23.334904 20133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:23.334947 20133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:08:23.336318 20133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:23.336591 20133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:23.336884 20133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:23.342528 20138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.342563 20141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.342563 20139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:23.343001 20133 server_base.cc:1034] running on GCE node
I20260928 09:08:23.343232 20133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:23.343482 20133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:23.344643 20133 hybrid_clock.cc:648] HybridClock initialized: now 1790586503344630 us; error 31 us; skew 500 ppm
I20260928 09:08:23.345965 20133 webserver.cc:460] Webserver started at http://127.0.98.1:39417/ using document root <none> and password file <none>
I20260928 09:08:23.346218 20133 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:23.346298 20133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:23.346447 20133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:23.347579 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data/instance:
uuid: "eba85d00f5da463baac8c5498e3e5fad"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "0c60dc6fda4a0fe508ab7b2ff487c4ff"
server_key_iv: "0f011ef44f7cabd2281f3b1003fdd585"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.347956 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eba85d00f5da463baac8c5498e3e5fad"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "0c60dc6fda4a0fe508ab7b2ff487c4ff"
server_key_iv: "0f011ef44f7cabd2281f3b1003fdd585"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.349357 20133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 09:08:23.350206 20147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.350381 20133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:08:23.350483 20133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal
uuid: "eba85d00f5da463baac8c5498e3e5fad"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "0c60dc6fda4a0fe508ab7b2ff487c4ff"
server_key_iv: "0f011ef44f7cabd2281f3b1003fdd585"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.350569 20133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:23.360850 20133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:23.361163 20133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:23.361327 20133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:23.361622 20133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:23.361966 20133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:23.362020 20133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.362054 20133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:23.362083 20133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.367805 20133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44913
I20260928 09:08:23.367846 20260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44913 every 8 connection(s)
I20260928 09:08:23.368189 20133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:08:23.373780 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.369067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37457 (local address 127.0.98.62:42793)
0928 09:08:23.369147 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.369150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.369439 (+ 289us) server_negotiation.cc:407] Connection header received
0928 09:08:23.369968 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.369969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.369984 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.370041 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:23.370789 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.371303 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.372275 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.372462 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.372517 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.373439 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:23.373440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:23.373440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:23.373444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:23.373457 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:23.373461 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:23.373503 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.373653 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.373698 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:08:23.374019 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.369074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.98.1:37457)
0928 09:08:23.369301 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.369341 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.369868 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.370118 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.370122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.370131 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:23.370697 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.370701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.371388 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.371389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.372187 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.372189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.372559 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.372561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.372689 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.373283 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:23.373284 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:23.373314 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:23.373339 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:23.373519 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:23.373524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:23.373526 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.373651 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.373756 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260928 09:08:23.374603 20261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42793
I20260928 09:08:23.374686 20261 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:23.374831 20261 heartbeater.cc:511] Master 127.0.98.62:42793 requested a full tablet report, sending...
I20260928 09:08:23.375362 20073 ts_manager.cc:194] Registered new tserver with Master: eba85d00f5da463baac8c5498e3e5fad (127.0.98.1:44913)
I20260928 09:08:23.375437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20133
I20260928 09:08:23.375530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:08:23.375763 392 external_mini_cluster.cc:1428] Setting key 264af645f06025cf22815105deadeed5
I20260928 09:08:23.376382 20073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37457
I20260928 09:08:23.377007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42793
--builtin_ntp_servers=127.0.98.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:08:23.461711 20264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:23.461905 20264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:23.461926 20264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:23.463065 20264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:23.463102 20264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:23.463127 20264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:08:23.464488 20264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:23.464676 20264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:23.464919 20264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:23.469863 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.469895 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.469869 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:23.470077 20264 server_base.cc:1034] running on GCE node
I20260928 09:08:23.470335 20264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:23.470548 20264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:23.471692 20264 hybrid_clock.cc:648] HybridClock initialized: now 1790586503471679 us; error 26 us; skew 500 ppm
I20260928 09:08:23.472972 20264 webserver.cc:460] Webserver started at http://127.0.98.2:43931/ using document root <none> and password file <none>
I20260928 09:08:23.473239 20264 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:23.473311 20264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:23.473471 20264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:23.474668 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data/instance:
uuid: "2479ae0ed5534d58b676acb56c6a10da"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "35912c19e3b482bb53decc113b271b36"
server_key_iv: "908f5b627f385e15789776d58ef74925"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.475032 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2479ae0ed5534d58b676acb56c6a10da"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "35912c19e3b482bb53decc113b271b36"
server_key_iv: "908f5b627f385e15789776d58ef74925"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.476315 20264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:23.477079 20278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.477250 20264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:23.477347 20264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal
uuid: "2479ae0ed5534d58b676acb56c6a10da"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "35912c19e3b482bb53decc113b271b36"
server_key_iv: "908f5b627f385e15789776d58ef74925"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.477427 20264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:23.493394 20264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:23.493784 20264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:23.493940 20264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:23.494177 20264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:23.494514 20264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:23.494568 20264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.494606 20264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:23.494634 20264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.500361 20264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39583
I20260928 09:08:23.500419 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39583 every 8 connection(s)
I20260928 09:08:23.500785 20264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:08:23.501868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20264
I20260928 09:08:23.501987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:08:23.502198 392 external_mini_cluster.cc:1428] Setting key 1fbb0633c99ea89179f4e63b110d311c
I20260928 09:08:23.503201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42793
--builtin_ntp_servers=127.0.98.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 09:08:23.506829 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.501578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56215 (local address 127.0.98.62:42793)
0928 09:08:23.501656 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.501658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.501935 (+ 277us) server_negotiation.cc:407] Connection header received
0928 09:08:23.502695 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.502697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.502706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.502757 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:23.503563 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.504068 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.505075 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.505375 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.505437 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.506493 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:23.506495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:23.506495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:23.506501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:23.506518 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:23.506524 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:23.506588 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.506672 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.506703 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:08:23.507019 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.501549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.98.2:56215)
0928 09:08:23.501803 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.501837 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.502563 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.502837 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.502841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.502852 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:23.503470 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.503475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.504167 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.504168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.504971 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.504973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.505469 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.505470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.505647 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.506288 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:23.506289 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:23.506318 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:23.506365 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:23.506593 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:23.506597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:23.506599 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.506654 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.506766 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260928 09:08:23.507565 20392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42793
I20260928 09:08:23.507658 20392 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:23.507862 20392 heartbeater.cc:511] Master 127.0.98.62:42793 requested a full tablet report, sending...
I20260928 09:08:23.508257 20073 ts_manager.cc:194] Registered new tserver with Master: 2479ae0ed5534d58b676acb56c6a10da (127.0.98.2:39583)
I20260928 09:08:23.508836 20073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56215
W20260928 09:08:23.586015 20395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:23.586226 20395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:23.586257 20395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:23.587394 20395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:23.587432 20395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:23.587458 20395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:08:23.588868 20395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:23.589068 20395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:23.589361 20395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:23.594494 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.594523 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:23.594585 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:23.595136 20395 server_base.cc:1034] running on GCE node
I20260928 09:08:23.595386 20395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:23.595642 20395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:23.596823 20395 hybrid_clock.cc:648] HybridClock initialized: now 1790586503596809 us; error 32 us; skew 500 ppm
I20260928 09:08:23.598143 20395 webserver.cc:460] Webserver started at http://127.0.98.3:41791/ using document root <none> and password file <none>
I20260928 09:08:23.598389 20395 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:23.598436 20395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:23.598601 20395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:23.599679 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data/instance:
uuid: "13285e073fdc4dbabbd59e94b72eceaf"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "41c50525e2655a0a8f87f9a0f7e50a96"
server_key_iv: "db03060b2fd89e0c6d1e932affd784f5"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.600060 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13285e073fdc4dbabbd59e94b72eceaf"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "41c50525e2655a0a8f87f9a0f7e50a96"
server_key_iv: "db03060b2fd89e0c6d1e932affd784f5"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.601414 20395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:08:23.602272 20409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.602509 20395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:23.602603 20395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal
uuid: "13285e073fdc4dbabbd59e94b72eceaf"
format_stamp: "Formatted at 2026-09-28 09:08:23 on dist-test-slave-smns"
server_key: "41c50525e2655a0a8f87f9a0f7e50a96"
server_key_iv: "db03060b2fd89e0c6d1e932affd784f5"
server_key_version: "encryptionkey@0"
I20260928 09:08:23.602681 20395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:23.613740 20395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:23.614028 20395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:23.614168 20395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:23.614391 20395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:08:23.614705 20395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:08:23.614746 20395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.614789 20395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:08:23.614815 20395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:23.620821 20395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36627
I20260928 09:08:23.620864 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36627 every 8 connection(s)
I20260928 09:08:23.621181 20395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:08:23.626569 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.621938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40461 (local address 127.0.98.62:42793)
0928 09:08:23.622003 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.622005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.622230 (+ 225us) server_negotiation.cc:407] Connection header received
0928 09:08:23.622703 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.622704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.622713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.622789 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:23.623497 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.623995 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.625013 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.625194 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.625247 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.626271 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:23.626272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:23.626273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:23.626277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:23.626294 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:23.626303 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:23.626370 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.626459 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.626486 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:08:23.626865 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.621915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.98.3:40461)
0928 09:08:23.622097 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.622146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.622621 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.622850 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.622854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.622862 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:23.623421 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.623427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.624098 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.624099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.624937 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.624940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.625270 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.625270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.625378 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.626091 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:08:23.626092 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:23.626129 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:08:23.626158 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:23.626372 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:23.626379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:23.626382 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.626460 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.626564 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 09:08:23.627408 20523 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42793
I20260928 09:08:23.627529 20523 heartbeater.cc:464] Registering TS with master...
I20260928 09:08:23.627740 20523 heartbeater.cc:511] Master 127.0.98.62:42793 requested a full tablet report, sending...
I20260928 09:08:23.628254 20073 ts_manager.cc:194] Registered new tserver with Master: 13285e073fdc4dbabbd59e94b72eceaf (127.0.98.3:36627)
I20260928 09:08:23.628270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20395
I20260928 09:08:23.628381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:08:23.628602 392 external_mini_cluster.cc:1428] Setting key 6bef2f0fc84f7020a5add38addcf20bc
I20260928 09:08:23.628870 20073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40461
I20260928 09:08:23.630106 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:08:23.634336 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.631463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.0.98.62:42793)
0928 09:08:23.631530 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.631532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.631654 (+ 122us) server_negotiation.cc:407] Connection header received
0928 09:08:23.631695 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.631696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.631705 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.631761 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:23.631891 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.632387 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.633140 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.633335 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.633385 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.634095 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:23.634096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:23.634096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:23.634100 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:23.634113 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:23.634116 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:23.634159 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.634260 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.634280 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 09:08:23.635936 20072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:08:23.637492 20072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:08:23.643890 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.639609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44913 (local address 127.0.0.1:56500)
0928 09:08:23.639919 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.639961 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.640055 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.640436 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.640436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.640438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:23.640472 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.640474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.641769 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.641770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.642831 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.642833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.643646 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.643646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.643732 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.643735 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.643793 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.643820 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":23,"thread_start_us":59,"threads_started":1}
I20260928 09:08:23.643919 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.639703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.0.98.1:44913)
0928 09:08:23.639948 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.639953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.640024 (+ 71us) server_negotiation.cc:407] Connection header received
0928 09:08:23.640276 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.640279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.640298 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.640372 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:23.640534 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.641672 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.642901 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.643574 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.643637 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.643709 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.643807 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.643843 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260928 09:08:23.643890 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.639628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39583 (local address 127.0.0.1:59218)
0928 09:08:23.639941 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.639974 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.640035 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.640370 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.640371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.640373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:23.640437 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.640439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.641348 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.641349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.642640 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.642643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.643565 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.643566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.643652 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.643668 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.643743 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.643805 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":96,"threads_started":1}
I20260928 09:08:23.644709 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.639820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36627 (local address 127.0.0.1:43008)
0928 09:08:23.640098 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.640107 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.640154 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.640360 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.640363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.640365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:23.640431 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.640433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.641792 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.641793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.642650 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.642652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.644482 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.644483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.644539 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.644543 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.644589 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.644635 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":67,"threads_started":1}
I20260928 09:08:23.644819 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.639727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.0.98.2:39583)
0928 09:08:23.640034 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.640038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.640048 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:08:23.640167 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.640170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.640187 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.640291 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:23.640517 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.641256 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.642744 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.643478 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.643561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.643658 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.643936 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.644733 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":86,"threads_started":1}
I20260928 09:08:23.644802 20195 tablet_service.cc:1467] Processing CreateTablet for tablet b1a5010b6cfb41709fb5d501198fcf1e (DEFAULT_TABLE table=test-table [id=5f50642536754ff2a0c0fa298d69211f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:23.645025 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.639907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.98.3:36627)
0928 09:08:23.640088 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.640093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.640214 (+ 121us) server_negotiation.cc:407] Connection header received
0928 09:08:23.640250 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.640252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.640264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.640613 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:23.640665 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.641724 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.643696 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.644383 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.644478 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.644874 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.644899 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.644941 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260928 09:08:23.645186 20195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a5010b6cfb41709fb5d501198fcf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:23.645430 20326 tablet_service.cc:1467] Processing CreateTablet for tablet b1a5010b6cfb41709fb5d501198fcf1e (DEFAULT_TABLE table=test-table [id=5f50642536754ff2a0c0fa298d69211f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:23.645780 20326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a5010b6cfb41709fb5d501198fcf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:23.648701 20537 tablet_bootstrap.cc:492] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad: Bootstrap starting.
I20260928 09:08:23.649209 20538 tablet_bootstrap.cc:492] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Bootstrap starting.
I20260928 09:08:23.649466 20537 tablet_bootstrap.cc:654] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:23.649686 20457 tablet_service.cc:1467] Processing CreateTablet for tablet b1a5010b6cfb41709fb5d501198fcf1e (DEFAULT_TABLE table=test-table [id=5f50642536754ff2a0c0fa298d69211f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:08:23.649818 20537 log.cc:824] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:23.649878 20538 tablet_bootstrap.cc:654] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:23.650041 20457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a5010b6cfb41709fb5d501198fcf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:23.650202 20538 log.cc:824] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:23.650652 20537 tablet_bootstrap.cc:492] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad: No bootstrap required, opened a new log
I20260928 09:08:23.650723 20537 ts_tablet_manager.cc:1397] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:23.650947 20538 tablet_bootstrap.cc:492] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: No bootstrap required, opened a new log
I20260928 09:08:23.651010 20538 ts_tablet_manager.cc:1397] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:23.652746 20537 raft_consensus.cc:348] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.652904 20537 raft_consensus.cc:374] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:23.652948 20537 raft_consensus.cc:729] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba85d00f5da463baac8c5498e3e5fad, State: Initialized, Role: FOLLOWER
I20260928 09:08:23.653112 20537 consensus_queue.cc:260] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.653407 20537 ts_tablet_manager.cc:1428] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:08:23.653532 20261 heartbeater.cc:503] Master 127.0.98.62:42793 was elected leader, sending a full tablet report...
I20260928 09:08:23.653565 20541 tablet_bootstrap.cc:492] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Bootstrap starting.
I20260928 09:08:23.653671 20538 raft_consensus.cc:348] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.653833 20538 raft_consensus.cc:374] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:23.653873 20538 raft_consensus.cc:729] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2479ae0ed5534d58b676acb56c6a10da, State: Initialized, Role: FOLLOWER
I20260928 09:08:23.654012 20538 consensus_queue.cc:260] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.654276 20541 tablet_bootstrap.cc:654] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:23.654276 20538 ts_tablet_manager.cc:1428] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:08:23.654311 20392 heartbeater.cc:503] Master 127.0.98.62:42793 was elected leader, sending a full tablet report...
I20260928 09:08:23.654616 20541 log.cc:824] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:23.655386 20541 tablet_bootstrap.cc:492] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: No bootstrap required, opened a new log
I20260928 09:08:23.655441 20541 ts_tablet_manager.cc:1397] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:08:23.657194 20541 raft_consensus.cc:348] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.657321 20541 raft_consensus.cc:374] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:23.657344 20541 raft_consensus.cc:729] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13285e073fdc4dbabbd59e94b72eceaf, State: Initialized, Role: FOLLOWER
I20260928 09:08:23.657461 20541 consensus_queue.cc:260] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.657761 20541 ts_tablet_manager.cc:1428] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 09:08:23.657809 20523 heartbeater.cc:503] Master 127.0.98.62:42793 was elected leader, sending a full tablet report...
I20260928 09:08:23.690690 20542 raft_consensus.cc:482] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:08:23.690821 20542 raft_consensus.cc:504] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.691308 20542 leader_election.cc:288] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2479ae0ed5534d58b676acb56c6a10da (127.0.98.2:39583), 13285e073fdc4dbabbd59e94b72eceaf (127.0.98.3:36627)
I20260928 09:08:23.694187 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.691610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41225 (local address 127.0.98.3:36627)
0928 09:08:23.691679 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.691681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.691690 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:23.691729 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.691730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.691737 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.691808 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:23.691977 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.692465 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.693412 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.693829 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.693912 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.693923 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.694074 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.694109 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 09:08:23.694187 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.691513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36627 (local address 127.0.98.1:41225)
0928 09:08:23.691572 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.691585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.691630 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.691812 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.691813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.691814 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:23.691898 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.691900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.692557 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.692559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.693311 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.693313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.693910 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.693912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.693985 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.693988 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.694065 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.694109 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 09:08:23.694294 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.691542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39583 (local address 127.0.98.1:56499)
0928 09:08:23.691730 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.691756 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:23.691833 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:23.691986 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:23.691986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:23.691989 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:08:23.692067 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.692069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.692721 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.692722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.693455 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:23.693456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.694018 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.694019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:23.694098 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.694101 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:08:23.694169 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:08:23.694210 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":120,"threads_started":1}
I20260928 09:08:23.694288 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.691583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56499 (local address 127.0.98.2:39583)
0928 09:08:23.691648 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.691650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.691820 (+ 170us) server_negotiation.cc:407] Connection header received
0928 09:08:23.691893 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.691894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.691905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.691980 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:08:23.692147 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.692627 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.693577 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.693929 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.694017 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.694026 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.694175 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.694212 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 09:08:23.694557 20477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a5010b6cfb41709fb5d501198fcf1e" candidate_uuid: "eba85d00f5da463baac8c5498e3e5fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13285e073fdc4dbabbd59e94b72eceaf" is_pre_election: true
I20260928 09:08:23.694571 20346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a5010b6cfb41709fb5d501198fcf1e" candidate_uuid: "eba85d00f5da463baac8c5498e3e5fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2479ae0ed5534d58b676acb56c6a10da" is_pre_election: true
I20260928 09:08:23.694738 20477 raft_consensus.cc:2454] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba85d00f5da463baac8c5498e3e5fad in term 0.
I20260928 09:08:23.694737 20346 raft_consensus.cc:2454] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba85d00f5da463baac8c5498e3e5fad in term 0.
I20260928 09:08:23.695012 20149 leader_election.cc:302] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13285e073fdc4dbabbd59e94b72eceaf, 2479ae0ed5534d58b676acb56c6a10da, eba85d00f5da463baac8c5498e3e5fad; no voters:
I20260928 09:08:23.695147 20542 raft_consensus.cc:2781] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:08:23.695210 20542 raft_consensus.cc:482] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:08:23.695240 20542 raft_consensus.cc:3037] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:23.695909 20542 raft_consensus.cc:504] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.696060 20542 leader_election.cc:288] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [CANDIDATE]: Term 1 election: Requested vote from peers 2479ae0ed5534d58b676acb56c6a10da (127.0.98.2:39583), 13285e073fdc4dbabbd59e94b72eceaf (127.0.98.3:36627)
I20260928 09:08:23.696234 20346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a5010b6cfb41709fb5d501198fcf1e" candidate_uuid: "eba85d00f5da463baac8c5498e3e5fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2479ae0ed5534d58b676acb56c6a10da"
I20260928 09:08:23.696228 20477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a5010b6cfb41709fb5d501198fcf1e" candidate_uuid: "eba85d00f5da463baac8c5498e3e5fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13285e073fdc4dbabbd59e94b72eceaf"
I20260928 09:08:23.696311 20477 raft_consensus.cc:3037] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:23.696311 20346 raft_consensus.cc:3037] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:23.697228 20346 raft_consensus.cc:2454] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba85d00f5da463baac8c5498e3e5fad in term 1.
I20260928 09:08:23.697247 20477 raft_consensus.cc:2454] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba85d00f5da463baac8c5498e3e5fad in term 1.
I20260928 09:08:23.697404 20151 leader_election.cc:302] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2479ae0ed5534d58b676acb56c6a10da, eba85d00f5da463baac8c5498e3e5fad; no voters:
I20260928 09:08:23.697518 20542 raft_consensus.cc:2781] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:23.697713 20542 raft_consensus.cc:686] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [term 1 LEADER]: Becoming Leader. State: Replica: eba85d00f5da463baac8c5498e3e5fad, State: Running, Role: LEADER
I20260928 09:08:23.697814 20542 consensus_queue.cc:237] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } }
I20260928 09:08:23.698642 20072 catalog_manager.cc:5495] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad reported cstate change: term changed from 0 to 1, leader changed from <none> to eba85d00f5da463baac8c5498e3e5fad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "eba85d00f5da463baac8c5498e3e5fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba85d00f5da463baac8c5498e3e5fad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 } health_report { overall_health: UNKNOWN } } }
I20260928 09:08:23.716404 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.714190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.98.1:44913)
0928 09:08:23.714233 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:23.714257 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:23.714271 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:08:23.714307 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:23.714308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:23.714326 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:23.714388 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:23.714488 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.714940 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.715723 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:23.715904 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:23.715955 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:23.716087 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:23.716182 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:23.716261 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:23.716307 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 09:08:23.716326 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:23.718343 20346 raft_consensus.cc:1261] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 1 FOLLOWER]: Refusing update from remote peer eba85d00f5da463baac8c5498e3e5fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:23.718343 20477 raft_consensus.cc:1261] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 1 FOLLOWER]: Refusing update from remote peer eba85d00f5da463baac8c5498e3e5fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:23.718745 20542 consensus_queue.cc:1035] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "13285e073fdc4dbabbd59e94b72eceaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:23.718900 20547 consensus_queue.cc:1035] T b1a5010b6cfb41709fb5d501198fcf1e P eba85d00f5da463baac8c5498e3e5fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2479ae0ed5534d58b676acb56c6a10da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:23.721227 20550 mvcc.cc:205] Tried to move back new op lower bound from 7334242319227510784 to 7334242319146909696. Current Snapshot: MvccSnapshot[applied={T|T < 7334242319227510784}]
I20260928 09:08:23.721903 20552 mvcc.cc:205] Tried to move back new op lower bound from 7334242319227510784 to 7334242319146909696. Current Snapshot: MvccSnapshot[applied={T|T < 7334242319227510784}]
I20260928 09:08:23.722112 20551 mvcc.cc:205] Tried to move back new op lower bound from 7334242319227510784 to 7334242319146909696. Current Snapshot: MvccSnapshot[applied={T|T < 7334242319227510784 or (T in {7334242319227510784})}]
I20260928 09:08:23.723455 20072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:23.723551 20072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:08:23.724272 20072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75c7df09822486baa93d2581d81b3df: Sending DeleteTablet for 3 replicas of tablet b1a5010b6cfb41709fb5d501198fcf1e
I20260928 09:08:23.724786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20133
I20260928 09:08:23.724897 20457 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a5010b6cfb41709fb5d501198fcf1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:23 UTC) from {username='slave'} at 127.0.0.1:43008
I20260928 09:08:23.725761 20326 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a5010b6cfb41709fb5d501198fcf1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:08:23 UTC) from {username='slave'} at 127.0.0.1:59218
I20260928 09:08:23.726140 20561 tablet_replica.cc:330] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: stopping tablet replica
I20260928 09:08:23.726276 20561 raft_consensus.cc:2229] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:23.726410 20561 raft_consensus.cc:2258] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:23.726516 20560 tablet_replica.cc:330] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: stopping tablet replica
I20260928 09:08:23.726621 20560 raft_consensus.cc:2229] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:08:23.726737 20560 raft_consensus.cc:2258] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:08:23.726820 20561 ts_tablet_manager.cc:1905] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:23.727175 20560 ts_tablet_manager.cc:1905] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:08:23.728857 20561 ts_tablet_manager.cc:1918] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:23.728942 20561 log.cc:1196] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-1/wal/wals/b1a5010b6cfb41709fb5d501198fcf1e
I20260928 09:08:23.729130 20560 ts_tablet_manager.cc:1918] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:08:23.729203 20560 log.cc:1196] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Deleting WAL directory at /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790586261566264-392-0/minicluster-data/ts-2/wal/wals/b1a5010b6cfb41709fb5d501198fcf1e
I20260928 09:08:23.729257 20561 ts_tablet_manager.cc:1939] T b1a5010b6cfb41709fb5d501198fcf1e P 2479ae0ed5534d58b676acb56c6a10da: Deleting consensus metadata
I20260928 09:08:23.729486 20560 ts_tablet_manager.cc:1939] T b1a5010b6cfb41709fb5d501198fcf1e P 13285e073fdc4dbabbd59e94b72eceaf: Deleting consensus metadata
I20260928 09:08:23.729666 20060 catalog_manager.cc:4841] TS 2479ae0ed5534d58b676acb56c6a10da (127.0.98.2:39583): tablet b1a5010b6cfb41709fb5d501198fcf1e (table test-table [id=5f50642536754ff2a0c0fa298d69211f]) successfully deleted
I20260928 09:08:23.730293 20058 catalog_manager.cc:4841] TS 13285e073fdc4dbabbd59e94b72eceaf (127.0.98.3:36627): tablet b1a5010b6cfb41709fb5d501198fcf1e (table test-table [id=5f50642536754ff2a0c0fa298d69211f]) successfully deleted
W20260928 09:08:23.731186 20058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44913 (error 108)
I20260928 09:08:23.731431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20264
I20260928 09:08:23.732012 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:23.731499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44913 (local address 127.0.0.1:56518)
0928 09:08:23.731929 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:23.731973 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
W20260928 09:08:23.732231 20058 catalog_manager.cc:4584] TS eba85d00f5da463baac8c5498e3e5fad (127.0.98.1:44913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1a5010b6cfb41709fb5d501198fcf1e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44913: connect: Connection refused (error 111)
I20260928 09:08:23.736752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20395
I20260928 09:08:23.741144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20042
2026-09-28T09:08:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (663 ms)
[----------] 6 tests from EncryptionPolicyTest (8488 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:23 dist-test-slave-smns krb5kdc[20565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:23 dist-test-slave-smns krb5kdc[20565](info): set up 2 sockets
Sep 28 09:08:23 dist-test-slave-smns krb5kdc[20565](info): commencing operation
krb5kdc: starting...
W20260928 09:08:24.785513 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:24 dist-test-slave-smns krb5kdc[20565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:24.909372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:24.991806 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:24.992062 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:24.992105 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:24.993355 20578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:24.993417 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:24.993441 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:24.993475 20578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:24.993505 20578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:24.995080 20578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:24.995378 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:24.995615 20578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:25.000605 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:25.000669 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:25.000639 20578 server_base.cc:1034] running on GCE node
W20260928 09:08:25.000615 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:25.001192 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:25.001459 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:25.002628 20578 hybrid_clock.cc:648] HybridClock initialized: now 1790586505002612 us; error 27 us; skew 500 ppm
Sep 28 09:08:25 dist-test-slave-smns krb5kdc[20565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:25.005192 20578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:25.005744 20578 webserver.cc:460] Webserver started at http://127.0.0.1:44153/ using document root <none> and password file <none>
I20260928 09:08:25.005975 20578 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:25.006039 20578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:25.006178 20578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:25.007274 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "c31e39aefb9549258899f765faf8b878"
format_stamp: "Formatted at 2026-09-28 09:08:25 on dist-test-slave-smns"
server_key: "7b2cbfb4f61479994528dbaa1050c4bb"
server_key_iv: "9d908ca3c8c0c2f50102838b5469c4a3"
server_key_version: "encryptionkey@0"
I20260928 09:08:25.007665 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "c31e39aefb9549258899f765faf8b878"
format_stamp: "Formatted at 2026-09-28 09:08:25 on dist-test-slave-smns"
server_key: "7b2cbfb4f61479994528dbaa1050c4bb"
server_key_iv: "9d908ca3c8c0c2f50102838b5469c4a3"
server_key_version: "encryptionkey@0"
I20260928 09:08:25.008996 20578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:25.009804 20593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:25.009977 20578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:25.010075 20578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "c31e39aefb9549258899f765faf8b878"
format_stamp: "Formatted at 2026-09-28 09:08:25 on dist-test-slave-smns"
server_key: "7b2cbfb4f61479994528dbaa1050c4bb"
server_key_iv: "9d908ca3c8c0c2f50102838b5469c4a3"
server_key_version: "encryptionkey@0"
I20260928 09:08:25.010157 20578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:25.025584 20578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:25.027482 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:25.027649 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:25.031958 20578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40185
I20260928 09:08:25.032014 20645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40185 every 8 connection(s)
I20260928 09:08:25.032385 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:25.033646 20646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:25.034394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20578
I20260928 09:08:25.034487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:25.034683 392 external_mini_cluster.cc:1428] Setting key 5106959edc3e53b36f02f1803a7aee91
I20260928 09:08:25.036851 20646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878: Bootstrap starting.
I20260928 09:08:25.037463 20646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:25.037787 20646 log.cc:824] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:25.038475 20646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878: No bootstrap required, opened a new log
I20260928 09:08:25.040278 20646 raft_consensus.cc:348] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31e39aefb9549258899f765faf8b878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40185 } }
I20260928 09:08:25.040450 20646 raft_consensus.cc:374] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:25.040498 20646 raft_consensus.cc:729] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31e39aefb9549258899f765faf8b878, State: Initialized, Role: FOLLOWER
I20260928 09:08:25.040634 20646 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31e39aefb9549258899f765faf8b878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40185 } }
I20260928 09:08:25.040714 20646 raft_consensus.cc:388] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:25.040757 20646 raft_consensus.cc:482] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:25.040805 20646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 09:08:25 dist-test-slave-smns krb5kdc[20565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:08:25.041383 20646 raft_consensus.cc:504] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31e39aefb9549258899f765faf8b878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40185 } }
I20260928 09:08:25.041496 20646 leader_election.cc:302] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c31e39aefb9549258899f765faf8b878; no voters:
I20260928 09:08:25.041728 20646 leader_election.cc:288] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:25.041754 20651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:25.041877 20651 raft_consensus.cc:686] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [term 1 LEADER]: Becoming Leader. State: Replica: c31e39aefb9549258899f765faf8b878, State: Running, Role: LEADER
I20260928 09:08:25.041956 20651 consensus_queue.cc:237] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31e39aefb9549258899f765faf8b878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40185 } }
I20260928 09:08:25.042003 20646 sys_catalog.cc:564] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:25.042284 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e39aefb9549258899f765faf8b878. Latest consensus state: current_term: 1 leader_uuid: "c31e39aefb9549258899f765faf8b878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31e39aefb9549258899f765faf8b878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40185 } } }
I20260928 09:08:25.042367 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:25.042297 20652 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c31e39aefb9549258899f765faf8b878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31e39aefb9549258899f765faf8b878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40185 } } }
I20260928 09:08:25.042424 20652 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:25.043210 20656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:25.043802 20656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:25.045398 20656 catalog_manager.cc:1347] Generated new cluster ID: 0dfcfc73c0a240cf862556f8afc51354
I20260928 09:08:25.045452 20656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:25.046597 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:25.035277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47025 (local address 127.0.0.1:40185)
0928 09:08:25.035445 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:25.035453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:25.035479 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:08:25.035900 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:25.035907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:25.035954 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:25.036070 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:25.036915 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:25.037956 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:25.038781 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:25.039186 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:25.039321 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:25.042007 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:25.042046 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:25.042046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:25.042062 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:25.044379 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:25.044861 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:25.044864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:25.044864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:25.044967 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:25.045261 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:25.045262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:25.045263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:25.045600 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:25.046189 (+ 589us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:25.046290 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 09:08:25.046396 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260928 09:08:25.047504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:25.056442 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:25.049000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41791 (local address 127.0.0.1:40185)
0928 09:08:25.049033 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:25.049035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:25.049142 (+ 107us) server_negotiation.cc:407] Connection header received
0928 09:08:25.049171 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:25.049172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:25.049182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:25.049259 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:25.049920 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:25.050752 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:25.051789 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:25.052125 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:25.052202 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:25.053604 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:25.053605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:25.053606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:25.053613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:25.055194 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:25.055617 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:25.055618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:25.055619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:25.055684 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:25.055912 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:25.055913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:25.055913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:25.056048 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:25.056168 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:25.056279 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 09:08:25.056344 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:08:25.061354 20656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:25.061936 20656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:25.067878 20656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c31e39aefb9549258899f765faf8b878: Generated new TSK 0
I20260928 09:08:25.068081 20656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:25.072155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20578
W20260928 09:08:25.076632 20563 connection.cc:535] client connection to 127.0.0.1:40185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40185): Connection reset by peer (error 104)
2026-09-28T09:08:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:25 dist-test-slave-smns krb5kdc[20677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:25 dist-test-slave-smns krb5kdc[20677](info): set up 2 sockets
Sep 28 09:08:25 dist-test-slave-smns krb5kdc[20677](info): commencing operation
krb5kdc: starting...
W20260928 09:08:26.110436 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:26 dist-test-slave-smns krb5kdc[20677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:26.233320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:26.315482 20690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:26.315737 20690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:26.315761 20690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:26.316892 20690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:26.316931 20690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:26.316946 20690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:26.316962 20690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:26.316974 20690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:26.318501 20690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:26.318710 20690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:26.318881 20690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:26.323892 20696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:26.323925 20695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:26.323892 20698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:26.324133 20690 server_base.cc:1034] running on GCE node
I20260928 09:08:26.324400 20690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:26.324759 20690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:26.325927 20690 hybrid_clock.cc:648] HybridClock initialized: now 1790586506325916 us; error 41 us; skew 500 ppm
Sep 28 09:08:26 dist-test-slave-smns krb5kdc[20677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:26.335409 20690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:26.335875 20690 webserver.cc:460] Webserver started at http://127.0.0.1:45917/ using document root <none> and password file <none>
I20260928 09:08:26.336104 20690 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:26.336172 20690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:26.336328 20690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:26.337416 20690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "79d979a469444e24852cce2a291b1133"
format_stamp: "Formatted at 2026-09-28 09:08:26 on dist-test-slave-smns"
server_key: "0fef8949dafc96164583e35d2377069d"
server_key_iv: "af1960a7c66ab8871691d46de0be4f89"
server_key_version: "encryptionkey@0"
I20260928 09:08:26.337845 20690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "79d979a469444e24852cce2a291b1133"
format_stamp: "Formatted at 2026-09-28 09:08:26 on dist-test-slave-smns"
server_key: "0fef8949dafc96164583e35d2377069d"
server_key_iv: "af1960a7c66ab8871691d46de0be4f89"
server_key_version: "encryptionkey@0"
I20260928 09:08:26.339267 20690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:26.340030 20705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:26.340209 20690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:26.340312 20690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "79d979a469444e24852cce2a291b1133"
format_stamp: "Formatted at 2026-09-28 09:08:26 on dist-test-slave-smns"
server_key: "0fef8949dafc96164583e35d2377069d"
server_key_iv: "af1960a7c66ab8871691d46de0be4f89"
server_key_version: "encryptionkey@0"
I20260928 09:08:26.340407 20690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:26.363547 20690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:26.365971 20690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:26.366160 20690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:26.370684 20690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45129
I20260928 09:08:26.370755 20757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45129 every 8 connection(s)
I20260928 09:08:26.371165 20690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:26.372090 20758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:26.374486 20758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133: Bootstrap starting.
I20260928 09:08:26.375092 20758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:26.375382 20758 log.cc:824] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:26.376114 20758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133: No bootstrap required, opened a new log
I20260928 09:08:26.377840 20758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d979a469444e24852cce2a291b1133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260928 09:08:26.377985 20758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:26.378008 20758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d979a469444e24852cce2a291b1133, State: Initialized, Role: FOLLOWER
I20260928 09:08:26.378201 20758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d979a469444e24852cce2a291b1133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260928 09:08:26.378269 20758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:26.378291 20758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:26.378291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20690
I20260928 09:08:26.378355 20758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:26.378427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:26.378623 392 external_mini_cluster.cc:1428] Setting key 25c5a363f0d6bc3c6fa9c977095d2cb7
I20260928 09:08:26.379102 20758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d979a469444e24852cce2a291b1133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260928 09:08:26.379218 20758 leader_election.cc:302] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d979a469444e24852cce2a291b1133; no voters:
I20260928 09:08:26.379405 20758 leader_election.cc:288] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:26.379438 20763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:26.379549 20763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [term 1 LEADER]: Becoming Leader. State: Replica: 79d979a469444e24852cce2a291b1133, State: Running, Role: LEADER
I20260928 09:08:26.379640 20763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d979a469444e24852cce2a291b1133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260928 09:08:26.379671 20758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:26.380136 20764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79d979a469444e24852cce2a291b1133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d979a469444e24852cce2a291b1133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } } }
I20260928 09:08:26.380229 20764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:26.380451 20771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:26.380502 20765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d979a469444e24852cce2a291b1133. Latest consensus state: current_term: 1 leader_uuid: "79d979a469444e24852cce2a291b1133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d979a469444e24852cce2a291b1133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } } }
I20260928 09:08:26.380656 20765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:26.381021 20771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:26.382923 20771 catalog_manager.cc:1347] Generated new cluster ID: 1d93fe153926429591af46e45696651f
I20260928 09:08:26.382987 20771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:26.385423 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:26.379177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33559 (local address 127.0.0.1:45129)
0928 09:08:26.379420 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:26.379428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:26.379481 (+ 53us) server_negotiation.cc:407] Connection header received
0928 09:08:26.379620 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:26.379629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:26.379688 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:26.380125 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:26.380236 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:26.381941 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:26.383038 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:26.383741 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:26.383869 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:26.384652 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:26.384656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:26.384657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:26.384885 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:26.384927 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:26.384944 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:26.385046 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:26.385138 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:08:26.385170 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
I20260928 09:08:26.386605 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:26.392270 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:26.388388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43365 (local address 127.0.0.1:45129)
0928 09:08:26.388421 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:26.388423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:26.388650 (+ 227us) server_negotiation.cc:407] Connection header received
0928 09:08:26.388677 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:26.388678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:26.388688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:26.388753 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:26.388939 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:26.389744 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:26.390866 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:26.391200 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:26.391259 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:26.392004 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:26.392005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:26.392006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:26.392010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:26.392027 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:26.392031 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:26.392085 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:26.392183 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 09:08:26.392194 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 09:08:26.394421 20771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:26.395148 20771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:26.403291 20771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d979a469444e24852cce2a291b1133: Generated new TSK 0
I20260928 09:08:26.403491 20771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:26.405412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20690
W20260928 09:08:26.409837 20675 connection.cc:535] client connection to 127.0.0.1:45129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45129): Connection reset by peer (error 104)
2026-09-28T09:08:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:26 dist-test-slave-smns krb5kdc[20789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:26 dist-test-slave-smns krb5kdc[20789](info): set up 2 sockets
Sep 28 09:08:26 dist-test-slave-smns krb5kdc[20789](info): commencing operation
krb5kdc: starting...
W20260928 09:08:27.445732 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:27 dist-test-slave-smns krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:27.568812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:27.653687 20802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:27.653891 20802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:27.653914 20802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:27.655031 20802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:27.655067 20802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:27.655087 20802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:27.655100 20802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:27.655112 20802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:27.656541 20802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:27.656725 20802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:27.656885 20802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:27.661708 20810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:27.661711 20807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:27.661945 20808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:27.661947 20802 server_base.cc:1034] running on GCE node
I20260928 09:08:27.662247 20802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:27.662551 20802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:27.663722 20802 hybrid_clock.cc:648] HybridClock initialized: now 1790586507663698 us; error 41 us; skew 500 ppm
Sep 28 09:08:27 dist-test-slave-smns krb5kdc[20789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:27.666633 20802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:27.667169 20802 webserver.cc:460] Webserver started at http://127.0.0.1:45961/ using document root <none> and password file <none>
I20260928 09:08:27.667433 20802 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:27.667480 20802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:27.667649 20802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:27.668715 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "7ef74cc91da44c348fc0ae5315146f2a"
format_stamp: "Formatted at 2026-09-28 09:08:27 on dist-test-slave-smns"
server_key: "0f55f1fe254fe612451e1ba8d657c7cb"
server_key_iv: "c849707b264d90c7f6f5e92149022009"
server_key_version: "encryptionkey@0"
I20260928 09:08:27.669071 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ef74cc91da44c348fc0ae5315146f2a"
format_stamp: "Formatted at 2026-09-28 09:08:27 on dist-test-slave-smns"
server_key: "0f55f1fe254fe612451e1ba8d657c7cb"
server_key_iv: "c849707b264d90c7f6f5e92149022009"
server_key_version: "encryptionkey@0"
I20260928 09:08:27.670586 20802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:27.671397 20817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:27.671572 20802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:08:27.671666 20802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "7ef74cc91da44c348fc0ae5315146f2a"
format_stamp: "Formatted at 2026-09-28 09:08:27 on dist-test-slave-smns"
server_key: "0f55f1fe254fe612451e1ba8d657c7cb"
server_key_iv: "c849707b264d90c7f6f5e92149022009"
server_key_version: "encryptionkey@0"
I20260928 09:08:27.671782 20802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:27.675215 20802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:27.677532 20802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:27.677750 20802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:27.681713 20802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35939
I20260928 09:08:27.681777 20869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35939 every 8 connection(s)
I20260928 09:08:27.682147 20802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:27.683157 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:27.683216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20802
I20260928 09:08:27.683326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:27.683547 392 external_mini_cluster.cc:1428] Setting key 257fdbd40f65cc386f343182fc7dede1
I20260928 09:08:27.686131 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:27.684061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47445 (local address 127.0.0.1:35939)
0928 09:08:27.684288 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:27.684296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:27.684320 (+ 24us) server_negotiation.cc:407] Connection header received
0928 09:08:27.684775 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:27.684781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:27.684808 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:27.684923 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:27.685628 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:27.685632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:27.685632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:27.685638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:27.685688 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:27.685700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:27.685773 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:27.685859 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 09:08:27.685914 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260928 09:08:27.686643 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a: Bootstrap starting.
I20260928 09:08:27.687355 20870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:27.687428 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:27.687645 20870 log.cc:824] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:27.688324 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a: No bootstrap required, opened a new log
I20260928 09:08:27.689855 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:27.688895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43651 (local address 127.0.0.1:35939)
0928 09:08:27.688931 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:27.688933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:27.689002 (+ 69us) server_negotiation.cc:407] Connection header received
0928 09:08:27.689069 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:27.689069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:27.689073 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:27.689118 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:27.689633 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:27.689634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:27.689634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:27.689638 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:27.689652 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:27.689656 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:27.689743 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:27.689804 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 09:08:27.689814 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:08:27.690793 20870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef74cc91da44c348fc0ae5315146f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260928 09:08:27.690920 20870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:27.690943 20870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef74cc91da44c348fc0ae5315146f2a, State: Initialized, Role: FOLLOWER
I20260928 09:08:27.691082 20870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef74cc91da44c348fc0ae5315146f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260928 09:08:27.691159 20870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:27.691200 20870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:27.691251 20870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:27.691783 20870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef74cc91da44c348fc0ae5315146f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260928 09:08:27.691892 20870 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef74cc91da44c348fc0ae5315146f2a; no voters:
I20260928 09:08:27.692072 20870 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:27.692147 20880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:27.692324 20870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:27.692308 20880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef74cc91da44c348fc0ae5315146f2a, State: Running, Role: LEADER
I20260928 09:08:27.692461 20880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef74cc91da44c348fc0ae5315146f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260928 09:08:27.692899 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef74cc91da44c348fc0ae5315146f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef74cc91da44c348fc0ae5315146f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260928 09:08:27.692907 20882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef74cc91da44c348fc0ae5315146f2a. Latest consensus state: current_term: 1 leader_uuid: "7ef74cc91da44c348fc0ae5315146f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef74cc91da44c348fc0ae5315146f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260928 09:08:27.693028 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:27.693034 20882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:27.693253 20889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:27.693783 20889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:27.695309 20889 catalog_manager.cc:1347] Generated new cluster ID: ebfaed13524f4529935b50cbd943aa87
I20260928 09:08:27.695362 20889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:27.706177 20889 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:27.706364 20889 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:27.718202 20889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef74cc91da44c348fc0ae5315146f2a: Generated new TSK 0
I20260928 09:08:27.718422 20889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:27.724036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20802
2026-09-28T09:08:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:27 dist-test-slave-smns krb5kdc[20901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:27 dist-test-slave-smns krb5kdc[20901](info): set up 2 sockets
Sep 28 09:08:27 dist-test-slave-smns krb5kdc[20901](info): commencing operation
krb5kdc: starting...
W20260928 09:08:28.769660 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:28 dist-test-slave-smns krb5kdc[20901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:28.897254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:28.979810 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:28.980029 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:28.980054 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:28.981252 20914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:28.981302 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:28.981316 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:28.981329 20914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:28.981343 20914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:28.983503 20914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:28.983773 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:28.984021 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:28.989344 20920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:28.989372 20919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:28.989387 20914 server_base.cc:1034] running on GCE node
W20260928 09:08:28.989352 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:28.990024 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:28.990386 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:28.991568 20914 hybrid_clock.cc:648] HybridClock initialized: now 1790586508991557 us; error 38 us; skew 500 ppm
Sep 28 09:08:28 dist-test-slave-smns krb5kdc[20901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:28.994329 20914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:28.994805 20914 webserver.cc:460] Webserver started at http://127.0.0.1:44201/ using document root <none> and password file <none>
I20260928 09:08:28.995043 20914 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:28.995116 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:28.995271 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:28.996407 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0a564c2266744ccbaf725ae6c848c14c"
format_stamp: "Formatted at 2026-09-28 09:08:28 on dist-test-slave-smns"
server_key: "2267753c217f41c54ae7aeb20cad7a96"
server_key_iv: "0a044fdb09a85360ad78475027ce8f1e"
server_key_version: "encryptionkey@0"
I20260928 09:08:28.996836 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a564c2266744ccbaf725ae6c848c14c"
format_stamp: "Formatted at 2026-09-28 09:08:28 on dist-test-slave-smns"
server_key: "2267753c217f41c54ae7aeb20cad7a96"
server_key_iv: "0a044fdb09a85360ad78475027ce8f1e"
server_key_version: "encryptionkey@0"
I20260928 09:08:28.998312 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:28.999253 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:28.999483 20914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:28.999595 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0a564c2266744ccbaf725ae6c848c14c"
format_stamp: "Formatted at 2026-09-28 09:08:28 on dist-test-slave-smns"
server_key: "2267753c217f41c54ae7aeb20cad7a96"
server_key_iv: "0a044fdb09a85360ad78475027ce8f1e"
server_key_version: "encryptionkey@0"
I20260928 09:08:28.999673 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:29.011094 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:29.013677 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:29.013849 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:29.017939 20914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34941
I20260928 09:08:29.017984 20981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34941 every 8 connection(s)
I20260928 09:08:29.018364 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:29.019239 20982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:29.021695 20982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c: Bootstrap starting.
I20260928 09:08:29.022396 20982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:29.022378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 20914
I20260928 09:08:29.022512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:29.022727 392 external_mini_cluster.cc:1428] Setting key 084d5f160b556bef60cd8498268750bc
I20260928 09:08:29.022756 20982 log.cc:824] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:29.023586 20982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c: No bootstrap required, opened a new log
I20260928 09:08:29.026302 20982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a564c2266744ccbaf725ae6c848c14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260928 09:08:29.026479 20982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:29.026523 20982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a564c2266744ccbaf725ae6c848c14c, State: Initialized, Role: FOLLOWER
I20260928 09:08:29.026698 20982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a564c2266744ccbaf725ae6c848c14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260928 09:08:29.026794 20982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:29.026839 20982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:29.026892 20982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:29.027537 20982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a564c2266744ccbaf725ae6c848c14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260928 09:08:29.027642 20982 leader_election.cc:302] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a564c2266744ccbaf725ae6c848c14c; no voters:
I20260928 09:08:29.027809 20982 leader_election.cc:288] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:29.027864 20987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:29.028028 20987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [term 1 LEADER]: Becoming Leader. State: Replica: 0a564c2266744ccbaf725ae6c848c14c, State: Running, Role: LEADER
I20260928 09:08:29.028072 20982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:29.028158 20987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a564c2266744ccbaf725ae6c848c14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260928 09:08:29.028532 20987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a564c2266744ccbaf725ae6c848c14c. Latest consensus state: current_term: 1 leader_uuid: "0a564c2266744ccbaf725ae6c848c14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a564c2266744ccbaf725ae6c848c14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } } }
I20260928 09:08:29.028656 20987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:29.028846 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a564c2266744ccbaf725ae6c848c14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a564c2266744ccbaf725ae6c848c14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } } }
I20260928 09:08:29.028937 20996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:29.028968 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:29.029609 20996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:08:29 dist-test-slave-smns krb5kdc[20901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:08:29.034832 20996 catalog_manager.cc:1347] Generated new cluster ID: 3babda240b97419f84ddb103991e9e31
I20260928 09:08:29.035045 20996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:29.035163 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:29.023279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52645 (local address 127.0.0.1:34941)
0928 09:08:29.023493 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:29.023503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:29.023536 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:08:29.024097 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:29.024107 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:29.024189 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:29.024347 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:29.024948 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:29.026439 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:29.027419 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:29.028065 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:29.028545 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:29.031309 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:29.031314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:29.031314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:29.031327 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:29.033373 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:29.033840 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:29.033844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:29.033844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:29.033940 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:29.034169 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:29.034170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:29.034170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:29.034472 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:29.034646 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:29.034804 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 09:08:29.034949 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20260928 09:08:29.036172 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:29.043769 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:29.037611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52451 (local address 127.0.0.1:34941)
0928 09:08:29.037703 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:29.037705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:29.037758 (+ 53us) server_negotiation.cc:407] Connection header received
0928 09:08:29.037832 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:29.037833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:29.037842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:29.037897 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:29.038423 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:29.038895 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:29.039620 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:29.039830 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:29.039916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:29.041120 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:29.041121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:29.041122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:29.041129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:29.042633 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:29.042979 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:29.042980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:29.042981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:29.043039 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:29.043245 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:29.043246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:29.043246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:29.043362 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:29.043487 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:29.043570 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 09:08:29.043656 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 09:08:29.063290 20996 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:29.064059 20996 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:29.070914 20996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a564c2266744ccbaf725ae6c848c14c: Generated new TSK 0
I20260928 09:08:29.071118 20996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:29.081624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 20914
2026-09-28T09:08:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:29 dist-test-slave-smns krb5kdc[21013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:29 dist-test-slave-smns krb5kdc[21013](info): set up 2 sockets
Sep 28 09:08:29 dist-test-slave-smns krb5kdc[21013](info): commencing operation
krb5kdc: starting...
W20260928 09:08:30.121865 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:30 dist-test-slave-smns krb5kdc[21013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:30.248306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:30.332283 21026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:30.332639 21026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:30.332693 21026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:30.333901 21026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:30.333948 21026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:30.333962 21026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:30.334002 21026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:30.334031 21026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:30.335489 21026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:30.335742 21026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:30.335994 21026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:30.341188 21034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:30.341216 21031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:30.341200 21032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:30.341219 21026 server_base.cc:1034] running on GCE node
I20260928 09:08:30.341691 21026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:30.341982 21026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:30.343137 21026 hybrid_clock.cc:648] HybridClock initialized: now 1790586510343141 us; error 82 us; skew 500 ppm
Sep 28 09:08:30 dist-test-slave-smns krb5kdc[21013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:30.345867 21026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:30.346355 21026 webserver.cc:460] Webserver started at http://127.0.0.1:44699/ using document root <none> and password file <none>
I20260928 09:08:30.346580 21026 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:30.346625 21026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:30.346786 21026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:30.348001 21026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0ea350f094884f5fa2126c2af1e53e47"
format_stamp: "Formatted at 2026-09-28 09:08:30 on dist-test-slave-smns"
server_key: "90320c053979104b491a7cf5243fcf3e"
server_key_iv: "ea1af2004818119fde4394e7c3e20d16"
server_key_version: "encryptionkey@0"
I20260928 09:08:30.348380 21026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ea350f094884f5fa2126c2af1e53e47"
format_stamp: "Formatted at 2026-09-28 09:08:30 on dist-test-slave-smns"
server_key: "90320c053979104b491a7cf5243fcf3e"
server_key_iv: "ea1af2004818119fde4394e7c3e20d16"
server_key_version: "encryptionkey@0"
I20260928 09:08:30.349906 21026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:08:30.350729 21041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:30.350925 21026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:30.351022 21026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0ea350f094884f5fa2126c2af1e53e47"
format_stamp: "Formatted at 2026-09-28 09:08:30 on dist-test-slave-smns"
server_key: "90320c053979104b491a7cf5243fcf3e"
server_key_iv: "ea1af2004818119fde4394e7c3e20d16"
server_key_version: "encryptionkey@0"
I20260928 09:08:30.351099 21026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:30.372905 21026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:30.375180 21026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:30.375353 21026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:30.379802 21026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45749
I20260928 09:08:30.379904 21093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45749 every 8 connection(s)
I20260928 09:08:30.380225 21026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:30.381103 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:30.383312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21026
I20260928 09:08:30.383440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:30.383594 392 external_mini_cluster.cc:1428] Setting key ba18262f13533a61633056df0e15e514
I20260928 09:08:30.383788 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47: Bootstrap starting.
I20260928 09:08:30.384527 21094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:30.384828 21094 log.cc:824] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:30.385528 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47: No bootstrap required, opened a new log
I20260928 09:08:30.387393 21094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea350f094884f5fa2126c2af1e53e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260928 09:08:30.387557 21094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:30.387604 21094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea350f094884f5fa2126c2af1e53e47, State: Initialized, Role: FOLLOWER
I20260928 09:08:30.387763 21094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea350f094884f5fa2126c2af1e53e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260928 09:08:30.387844 21094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:30.387904 21094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:30.387957 21094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:30.388686 21094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea350f094884f5fa2126c2af1e53e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260928 09:08:30.388813 21094 leader_election.cc:302] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ea350f094884f5fa2126c2af1e53e47; no voters:
I20260928 09:08:30.389022 21094 leader_election.cc:288] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:30.389047 21099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:30.389258 21099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea350f094884f5fa2126c2af1e53e47, State: Running, Role: LEADER
I20260928 09:08:30.389353 21094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:30.389415 21099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea350f094884f5fa2126c2af1e53e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260928 09:08:30.389801 21100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ea350f094884f5fa2126c2af1e53e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea350f094884f5fa2126c2af1e53e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } } }
I20260928 09:08:30.389886 21100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:30.390141 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:30.384152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35067 (local address 127.0.0.1:45749)
0928 09:08:30.384338 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:30.384347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:30.384382 (+ 35us) server_negotiation.cc:407] Connection header received
0928 09:08:30.384871 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:30.384878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:30.384952 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:30.385117 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:30.385434 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:30.386441 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:30.387261 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:30.387768 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:30.387879 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:30.388655 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:30.388659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:30.388660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:30.388861 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:30.388900 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:30.388931 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:30.388991 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:30.389058 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 09:08:30.389124 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 09:08:30.389905 21101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea350f094884f5fa2126c2af1e53e47. Latest consensus state: current_term: 1 leader_uuid: "0ea350f094884f5fa2126c2af1e53e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea350f094884f5fa2126c2af1e53e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } } }
I20260928 09:08:30.390573 21101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:30.390587 21108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:30.391224 21108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:30.391422 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:30.392940 21108 catalog_manager.cc:1347] Generated new cluster ID: 766a463fb94440b78f409d2c0ffffbb5
I20260928 09:08:30.392997 21108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:30.397207 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:30.393981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49963 (local address 127.0.0.1:45749)
0928 09:08:30.394024 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:30.394027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:30.394125 (+ 98us) server_negotiation.cc:407] Connection header received
0928 09:08:30.394314 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:30.394315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:30.394327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:30.394381 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:30.394551 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:30.395069 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:30.395875 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:30.396094 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:30.396182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:30.396923 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:30.396924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:30.396925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:30.396929 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:30.396945 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:30.396950 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:30.397024 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:30.397085 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 09:08:30.397109 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:30.426941 21108 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:30.427596 21108 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:30.430989 21108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ea350f094884f5fa2126c2af1e53e47: Generated new TSK 0
I20260928 09:08:30.431207 21108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:30.435381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21026
2026-09-28T09:08:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:30 dist-test-slave-smns krb5kdc[21125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:30 dist-test-slave-smns krb5kdc[21125](info): set up 2 sockets
Sep 28 09:08:30 dist-test-slave-smns krb5kdc[21125](info): commencing operation
krb5kdc: starting...
W20260928 09:08:31.501705 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:31 dist-test-slave-smns krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:31.628780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:31.716295 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:31.716519 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:31.716543 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:31.717798 21138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:31.717844 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:31.717860 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:31.717873 21138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:31.717885 21138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:31.719477 21138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:31.719688 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:31.719861 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:31.724699 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:31.724706 21144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:31.724718 21143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:31.725026 21138 server_base.cc:1034] running on GCE node
I20260928 09:08:31.725311 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:31.725667 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:31.726837 21138 hybrid_clock.cc:648] HybridClock initialized: now 1790586511726831 us; error 37 us; skew 500 ppm
Sep 28 09:08:31 dist-test-slave-smns krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:31.735261 21138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:31.735811 21138 webserver.cc:460] Webserver started at http://127.0.0.1:36061/ using document root <none> and password file <none>
I20260928 09:08:31.736053 21138 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:31.736122 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:31.736287 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:31.737380 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "f77e105b09c7439bb2dc78b4940a45f3"
format_stamp: "Formatted at 2026-09-28 09:08:31 on dist-test-slave-smns"
server_key: "e7a275492a0499f91036f8da3f85ce46"
server_key_iv: "4db23fa23c5c7252629dc160bfff3cc2"
server_key_version: "encryptionkey@0"
I20260928 09:08:31.737780 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "f77e105b09c7439bb2dc78b4940a45f3"
format_stamp: "Formatted at 2026-09-28 09:08:31 on dist-test-slave-smns"
server_key: "e7a275492a0499f91036f8da3f85ce46"
server_key_iv: "4db23fa23c5c7252629dc160bfff3cc2"
server_key_version: "encryptionkey@0"
I20260928 09:08:31.739157 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:31.739944 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:31.740118 21138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:31.740227 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "f77e105b09c7439bb2dc78b4940a45f3"
format_stamp: "Formatted at 2026-09-28 09:08:31 on dist-test-slave-smns"
server_key: "e7a275492a0499f91036f8da3f85ce46"
server_key_iv: "4db23fa23c5c7252629dc160bfff3cc2"
server_key_version: "encryptionkey@0"
I20260928 09:08:31.740329 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:31.743737 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:31.745767 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:31.745921 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:31.749807 21138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43899
I20260928 09:08:31.750001 21205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43899 every 8 connection(s)
I20260928 09:08:31.750218 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:31.751258 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:31.753695 21206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3: Bootstrap starting.
I20260928 09:08:31.753744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21138
I20260928 09:08:31.753903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:31.754096 392 external_mini_cluster.cc:1428] Setting key cd885f63002eb3d33a1cd2f015afe46c
I20260928 09:08:31.754469 21206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:31.754925 21206 log.cc:824] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:31.755715 21206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3: No bootstrap required, opened a new log
I20260928 09:08:31.756865 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:31.754614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33915 (local address 127.0.0.1:43899)
0928 09:08:31.754830 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:31.754841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:31.754878 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:08:31.755552 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:31.755561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:31.755595 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:31.755750 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:31.756435 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:31.756439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:31.756439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:31.756443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:31.756487 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:31.756500 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:31.756592 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:31.756624 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 09:08:31.756664 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260928 09:08:31.758018 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:31.758275 21206 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77e105b09c7439bb2dc78b4940a45f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260928 09:08:31.758440 21206 raft_consensus.cc:374] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:31.758474 21206 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77e105b09c7439bb2dc78b4940a45f3, State: Initialized, Role: FOLLOWER
I20260928 09:08:31.758610 21206 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77e105b09c7439bb2dc78b4940a45f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260928 09:08:31.758708 21206 raft_consensus.cc:388] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:31.758739 21206 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:31.758800 21206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:31.759618 21206 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77e105b09c7439bb2dc78b4940a45f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260928 09:08:31.759742 21206 leader_election.cc:302] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f77e105b09c7439bb2dc78b4940a45f3; no voters:
I20260928 09:08:31.759903 21206 leader_election.cc:288] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:31.759927 21216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:31.760035 21216 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [term 1 LEADER]: Becoming Leader. State: Replica: f77e105b09c7439bb2dc78b4940a45f3, State: Running, Role: LEADER
I20260928 09:08:31.760159 21206 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:31.760170 21216 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77e105b09c7439bb2dc78b4940a45f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260928 09:08:31.760579 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77e105b09c7439bb2dc78b4940a45f3. Latest consensus state: current_term: 1 leader_uuid: "f77e105b09c7439bb2dc78b4940a45f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77e105b09c7439bb2dc78b4940a45f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } } }
I20260928 09:08:31.760708 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:31.760735 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f77e105b09c7439bb2dc78b4940a45f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77e105b09c7439bb2dc78b4940a45f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } } }
I20260928 09:08:31.760838 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:31.761173 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:31.759615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52453 (local address 127.0.0.1:43899)
0928 09:08:31.759656 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:31.759658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:31.759796 (+ 138us) server_negotiation.cc:407] Connection header received
0928 09:08:31.759820 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:31.759822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:31.759827 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:31.759876 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:31.760943 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:31.760945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:31.760945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:31.760950 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:31.760966 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:31.760970 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:31.761021 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:31.761080 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 09:08:31.761095 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 09:08:31.762122 21232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:31.762169 21232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:31.762236 21223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:31.762377 21223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:31.763837 21223 catalog_manager.cc:1347] Generated new cluster ID: ed40595a264f4b209bbf8d08a1f6cdc4
I20260928 09:08:31.763896 21223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:31.790527 21223 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:31.790741 21223 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:31.797809 21223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f77e105b09c7439bb2dc78b4940a45f3: Generated new TSK 0
I20260928 09:08:31.798033 21223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:31.805469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21138
2026-09-28T09:08:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:31 dist-test-slave-smns krb5kdc[21237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:31 dist-test-slave-smns krb5kdc[21237](info): set up 2 sockets
Sep 28 09:08:31 dist-test-slave-smns krb5kdc[21237](info): commencing operation
krb5kdc: starting...
W20260928 09:08:32.842948 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:32 dist-test-slave-smns krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:32.969312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:33.051915 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:33.052130 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:33.052155 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:33.053275 21250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:33.053315 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:33.053330 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:33.053344 21250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:33.053355 21250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:33.054996 21250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:33.055224 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:33.055411 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:33.060760 21258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:33.060731 21250 server_base.cc:1034] running on GCE node
W20260928 09:08:33.060748 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:33.060792 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:33.061168 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:33.061493 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:33.062691 21250 hybrid_clock.cc:648] HybridClock initialized: now 1790586513062663 us; error 50 us; skew 500 ppm
Sep 28 09:08:33 dist-test-slave-smns krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:33.065425 21250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:33.065941 21250 webserver.cc:460] Webserver started at http://127.0.0.1:42469/ using document root <none> and password file <none>
I20260928 09:08:33.066160 21250 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:33.066202 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:33.066298 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:33.067356 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "824325cb1905482fa5bcbeee8b9ff42a"
format_stamp: "Formatted at 2026-09-28 09:08:33 on dist-test-slave-smns"
server_key: "260ac1d1f414761c35b204fabb0073fe"
server_key_iv: "03aba5e7a134ba32579c3dc257d9564c"
server_key_version: "encryptionkey@0"
I20260928 09:08:33.067713 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "824325cb1905482fa5bcbeee8b9ff42a"
format_stamp: "Formatted at 2026-09-28 09:08:33 on dist-test-slave-smns"
server_key: "260ac1d1f414761c35b204fabb0073fe"
server_key_iv: "03aba5e7a134ba32579c3dc257d9564c"
server_key_version: "encryptionkey@0"
I20260928 09:08:33.069020 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:08:33.069890 21265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:33.070106 21250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:33.070220 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "824325cb1905482fa5bcbeee8b9ff42a"
format_stamp: "Formatted at 2026-09-28 09:08:33 on dist-test-slave-smns"
server_key: "260ac1d1f414761c35b204fabb0073fe"
server_key_iv: "03aba5e7a134ba32579c3dc257d9564c"
server_key_version: "encryptionkey@0"
I20260928 09:08:33.070307 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:33.084704 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:33.087105 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:33.087275 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:33.091255 21250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34633
I20260928 09:08:33.091310 21317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34633 every 8 connection(s)
I20260928 09:08:33.091667 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:33.092553 21318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:33.094437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21250
I20260928 09:08:33.094561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:33.094769 392 external_mini_cluster.cc:1428] Setting key 0c20ebfbde3e5c361f982ed0912a59d4
I20260928 09:08:33.095067 21318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a: Bootstrap starting.
I20260928 09:08:33.095796 21318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:33.096127 21318 log.cc:824] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:33.096863 21318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a: No bootstrap required, opened a new log
I20260928 09:08:33.099263 21318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824325cb1905482fa5bcbeee8b9ff42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260928 09:08:33.099449 21318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:33.099488 21318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824325cb1905482fa5bcbeee8b9ff42a, State: Initialized, Role: FOLLOWER
I20260928 09:08:33.099634 21318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824325cb1905482fa5bcbeee8b9ff42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260928 09:08:33.099712 21318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:33.099742 21318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:33.099790 21318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:33.100512 21318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824325cb1905482fa5bcbeee8b9ff42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260928 09:08:33.100652 21318 leader_election.cc:302] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824325cb1905482fa5bcbeee8b9ff42a; no voters:
I20260928 09:08:33.100854 21318 leader_election.cc:288] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:33.100925 21323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:33.101089 21323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [term 1 LEADER]: Becoming Leader. State: Replica: 824325cb1905482fa5bcbeee8b9ff42a, State: Running, Role: LEADER
I20260928 09:08:33.101142 21318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:33.101908 21323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824325cb1905482fa5bcbeee8b9ff42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
Sep 28 09:08:33 dist-test-slave-smns krb5kdc[21237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:08:33.102530 21324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824325cb1905482fa5bcbeee8b9ff42a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824325cb1905482fa5bcbeee8b9ff42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } } }
I20260928 09:08:33.102636 21324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:33.102766 21325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824325cb1905482fa5bcbeee8b9ff42a. Latest consensus state: current_term: 1 leader_uuid: "824325cb1905482fa5bcbeee8b9ff42a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824325cb1905482fa5bcbeee8b9ff42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } } }
I20260928 09:08:33.102901 21325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a [sys.catalog]: This master's current role is: LEADER
W20260928 09:08:33.104028 21337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:33.104125 21337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:33.104182 21338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:33.104319 21338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:33.106031 21338 catalog_manager.cc:1347] Generated new cluster ID: f475e074920a45e195a7153447e9155c
I20260928 09:08:33.106083 21338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:33.107537 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:33.095402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:34449 (local address 127.0.0.1:34633)
0928 09:08:33.095590 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:33.095599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:33.095632 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:08:33.096245 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:33.096255 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:33.096325 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:33.096457 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:33.097155 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:33.098582 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:33.099416 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:33.100055 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:33.100189 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:33.103489 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:33.103493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:33.103493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:33.103505 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:33.105727 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:33.106209 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:33.106212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:33.106213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:33.106321 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:33.106570 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:33.106571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:33.106572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:33.106892 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:33.107053 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:33.107193 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 09:08:33.107307 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260928 09:08:33.110541 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:33.115927 21338 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:33.116492 21338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:33.121279 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:33.112049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:36323 (local address 127.0.0.1:34633)
0928 09:08:33.112120 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:33.112122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:33.113740 (+ 1618us) server_negotiation.cc:407] Connection header received
0928 09:08:33.113803 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:33.113804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:33.113814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:33.113872 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:33.114572 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:33.115332 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:33.116392 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:33.116691 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:33.116763 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:33.118360 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:33.118361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:33.118361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:33.118369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:33.120022 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:33.120463 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:33.120464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:33.120464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:33.120518 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:33.120753 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:33.120754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:33.120754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:33.120879 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:33.120969 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:33.121119 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 09:08:33.121185 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:08:33.128086 21338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824325cb1905482fa5bcbeee8b9ff42a: Generated new TSK 0
I20260928 09:08:33.128294 21338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:33.134320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21250
W20260928 09:08:33.138511 21235 connection.cc:535] client connection to 127.0.0.1:34633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34633): Connection reset by peer (error 104)
2026-09-28T09:08:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:33 dist-test-slave-smns krb5kdc[21349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:33 dist-test-slave-smns krb5kdc[21349](info): set up 2 sockets
Sep 28 09:08:33 dist-test-slave-smns krb5kdc[21349](info): commencing operation
krb5kdc: starting...
W20260928 09:08:34.181707 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:34 dist-test-slave-smns krb5kdc[21349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:34.305734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:34.387719 21362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:34.387923 21362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:34.387946 21362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:34.389389 21362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:34.389447 21362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:34.389462 21362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:34.389475 21362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:34.389488 21362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:34.391060 21362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:34.391260 21362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:34.391424 21362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:34.396481 21370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:34.396489 21367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:34.396489 21368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:34.396519 21362 server_base.cc:1034] running on GCE node
I20260928 09:08:34.396899 21362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:34.397150 21362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:34.398314 21362 hybrid_clock.cc:648] HybridClock initialized: now 1790586514398292 us; error 38 us; skew 500 ppm
Sep 28 09:08:34 dist-test-slave-smns krb5kdc[21349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:34.400910 21362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:34.401477 21362 webserver.cc:460] Webserver started at http://127.0.0.1:37595/ using document root <none> and password file <none>
I20260928 09:08:34.401753 21362 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:34.401837 21362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:34.402000 21362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:34.403126 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "3f64ea595db64c9c836e16d440b3c3ed"
format_stamp: "Formatted at 2026-09-28 09:08:34 on dist-test-slave-smns"
server_key: "a70d8ca7b49cabf629fb8c3a4269da8a"
server_key_iv: "c28a27d3090b622fae0cfe4c4f798ffb"
server_key_version: "encryptionkey@0"
I20260928 09:08:34.403502 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f64ea595db64c9c836e16d440b3c3ed"
format_stamp: "Formatted at 2026-09-28 09:08:34 on dist-test-slave-smns"
server_key: "a70d8ca7b49cabf629fb8c3a4269da8a"
server_key_iv: "c28a27d3090b622fae0cfe4c4f798ffb"
server_key_version: "encryptionkey@0"
I20260928 09:08:34.404938 21362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:34.405763 21377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:34.405944 21362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:34.406056 21362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "3f64ea595db64c9c836e16d440b3c3ed"
format_stamp: "Formatted at 2026-09-28 09:08:34 on dist-test-slave-smns"
server_key: "a70d8ca7b49cabf629fb8c3a4269da8a"
server_key_iv: "c28a27d3090b622fae0cfe4c4f798ffb"
server_key_version: "encryptionkey@0"
I20260928 09:08:34.406134 21362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:34.428526 21362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:34.430795 21362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:34.430974 21362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:34.435037 21362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40853
I20260928 09:08:34.435081 21429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40853 every 8 connection(s)
I20260928 09:08:34.435472 21362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:34.436373 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:34.438692 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed: Bootstrap starting.
I20260928 09:08:34.439205 21430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:34.439448 21430 log.cc:824] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:34.440114 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed: No bootstrap required, opened a new log
I20260928 09:08:34.440889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21362
I20260928 09:08:34.440996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:34.441174 392 external_mini_cluster.cc:1428] Setting key 8d27a68d9eb681dc03d1a6106843f0a0
I20260928 09:08:34.442363 21430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f64ea595db64c9c836e16d440b3c3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40853 } }
I20260928 09:08:34.442531 21430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:34.442567 21430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f64ea595db64c9c836e16d440b3c3ed, State: Initialized, Role: FOLLOWER
I20260928 09:08:34.442734 21430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f64ea595db64c9c836e16d440b3c3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40853 } }
I20260928 09:08:34.442806 21430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:34.442852 21430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:34.442902 21430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:34.443624 21430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f64ea595db64c9c836e16d440b3c3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40853 } }
I20260928 09:08:34.443744 21430 leader_election.cc:302] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f64ea595db64c9c836e16d440b3c3ed; no voters:
I20260928 09:08:34.443894 21430 leader_election.cc:288] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:34.443958 21435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:34.444130 21430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:34.444258 21435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [term 1 LEADER]: Becoming Leader. State: Replica: 3f64ea595db64c9c836e16d440b3c3ed, State: Running, Role: LEADER
I20260928 09:08:34.444366 21435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f64ea595db64c9c836e16d440b3c3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40853 } }
I20260928 09:08:34.444734 21436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f64ea595db64c9c836e16d440b3c3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f64ea595db64c9c836e16d440b3c3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40853 } } }
I20260928 09:08:34.444727 21435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f64ea595db64c9c836e16d440b3c3ed. Latest consensus state: current_term: 1 leader_uuid: "3f64ea595db64c9c836e16d440b3c3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f64ea595db64c9c836e16d440b3c3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40853 } } }
I20260928 09:08:34.444814 21436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:34.444828 21435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:34.445044 21444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:34.445592 21444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:34.447196 21444 catalog_manager.cc:1347] Generated new cluster ID: 2935ad15ce44425d8741b5d68878cc32
I20260928 09:08:34.447244 21444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:34.448271 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:34.441719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:57423 (local address 127.0.0.1:40853)
0928 09:08:34.441910 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:34.441919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:34.441963 (+ 44us) server_negotiation.cc:407] Connection header received
0928 09:08:34.442327 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:34.442337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:34.442404 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:34.442554 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:34.442888 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:34.444640 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:34.445947 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:34.446556 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:34.446690 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:34.447497 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:34.447501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:34.447501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:34.447722 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:34.447759 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:34.447774 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:34.447865 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:34.448020 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 09:08:34.448049 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 09:08:34.449215 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:34.454241 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:34.450757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:58841 (local address 127.0.0.1:40853)
0928 09:08:34.450788 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:34.450790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:34.450798 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:34.450862 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:34.450863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:34.450878 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:34.450943 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:34.451082 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:34.451835 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:34.452854 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:34.453224 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:34.453298 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:34.453965 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:34.453966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:34.453967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:34.453971 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:34.453985 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:34.453989 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:34.454063 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:34.454146 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 09:08:34.454167 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:08:34.460793 21444 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:34.461294 21444 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:34.465142 21444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f64ea595db64c9c836e16d440b3c3ed: Generated new TSK 0
I20260928 09:08:34.465328 21444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:34.468266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21362
W20260928 09:08:34.472113 21347 connection.cc:535] client connection to 127.0.0.1:40853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40853): Connection reset by peer (error 104)
2026-09-28T09:08:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:34 dist-test-slave-smns krb5kdc[21461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:34 dist-test-slave-smns krb5kdc[21461](info): set up 2 sockets
Sep 28 09:08:34 dist-test-slave-smns krb5kdc[21461](info): commencing operation
krb5kdc: starting...
W20260928 09:08:35.509826 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:35 dist-test-slave-smns krb5kdc[21461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:35.637665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:35.722452 21474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:35.722666 21474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:35.722690 21474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:35.723873 21474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:35.723917 21474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:35.723932 21474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:35.723944 21474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:35.723955 21474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:35.725430 21474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:35.725719 21474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:35.725904 21474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:35.731009 21479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:35.731050 21480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:35.731009 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:35.731200 21474 server_base.cc:1034] running on GCE node
I20260928 09:08:35.731564 21474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:35.731843 21474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:35.733023 21474 hybrid_clock.cc:648] HybridClock initialized: now 1790586515732993 us; error 59 us; skew 500 ppm
Sep 28 09:08:35 dist-test-slave-smns krb5kdc[21461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:35.735766 21474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:35.736274 21474 webserver.cc:460] Webserver started at http://127.0.0.1:46477/ using document root <none> and password file <none>
I20260928 09:08:35.736509 21474 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:35.736584 21474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:35.736747 21474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:35.737946 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "15496b43b694421d8783c489bd40ed58"
format_stamp: "Formatted at 2026-09-28 09:08:35 on dist-test-slave-smns"
server_key: "54c844e915468513e9470ba679c405f6"
server_key_iv: "f2a266a6026cd47fe69eb60385dac89b"
server_key_version: "encryptionkey@0"
I20260928 09:08:35.738318 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "15496b43b694421d8783c489bd40ed58"
format_stamp: "Formatted at 2026-09-28 09:08:35 on dist-test-slave-smns"
server_key: "54c844e915468513e9470ba679c405f6"
server_key_iv: "f2a266a6026cd47fe69eb60385dac89b"
server_key_version: "encryptionkey@0"
I20260928 09:08:35.739773 21474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:35.740650 21489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:35.740836 21474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:35.740943 21474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "15496b43b694421d8783c489bd40ed58"
format_stamp: "Formatted at 2026-09-28 09:08:35 on dist-test-slave-smns"
server_key: "54c844e915468513e9470ba679c405f6"
server_key_iv: "f2a266a6026cd47fe69eb60385dac89b"
server_key_version: "encryptionkey@0"
I20260928 09:08:35.741029 21474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:35.744503 21474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:35.746997 21474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:35.747172 21474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:35.751307 21474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37729
I20260928 09:08:35.751353 21541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37729 every 8 connection(s)
I20260928 09:08:35.751714 21474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:35.752566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21474
I20260928 09:08:35.752731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:35.752938 392 external_mini_cluster.cc:1428] Setting key 7ee26ec33f6caf39c36d218c53ee2fdc
I20260928 09:08:35.753152 21542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:35.755649 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:35.753483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:51541 (local address 127.0.0.1:37729)
0928 09:08:35.753702 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:35.753714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:35.753744 (+ 30us) server_negotiation.cc:407] Connection header received
0928 09:08:35.754238 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:35.754245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:35.754270 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:35.754393 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:35.755110 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:35.755113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:35.755113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:35.755117 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:35.755221 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:35.755235 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:35.755293 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:35.755390 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 09:08:35.755448 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":119,"threads_started":1}
I20260928 09:08:35.756469 21542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58: Bootstrap starting.
I20260928 09:08:35.756587 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:35.757252 21542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:35.757617 21542 log.cc:824] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:35.758452 21542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58: No bootstrap required, opened a new log
I20260928 09:08:35.760203 21542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15496b43b694421d8783c489bd40ed58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37729 } }
I20260928 09:08:35.760336 21542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:35.760360 21542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15496b43b694421d8783c489bd40ed58, State: Initialized, Role: FOLLOWER
I20260928 09:08:35.760496 21542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15496b43b694421d8783c489bd40ed58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37729 } }
I20260928 09:08:35.760603 21542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:35.760643 21542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:35.760692 21542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:35.761258 21542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15496b43b694421d8783c489bd40ed58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37729 } }
I20260928 09:08:35.761373 21542 leader_election.cc:302] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15496b43b694421d8783c489bd40ed58; no voters:
I20260928 09:08:35.761618 21542 leader_election.cc:288] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:35.761662 21552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:35.761752 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:35.758185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:44371 (local address 127.0.0.1:37729)
0928 09:08:35.758258 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:35.758260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:35.760669 (+ 2409us) server_negotiation.cc:407] Connection header received
0928 09:08:35.760700 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:35.760701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:35.760708 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:35.760784 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:35.761466 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:35.761468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:35.761468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:35.761473 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:35.761487 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:35.761492 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:35.761577 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:35.761660 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 09:08:35.761676 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 09:08:35.761830 21552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [term 1 LEADER]: Becoming Leader. State: Replica: 15496b43b694421d8783c489bd40ed58, State: Running, Role: LEADER
I20260928 09:08:35.761921 21542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:35.761904 21552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15496b43b694421d8783c489bd40ed58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37729 } }
I20260928 09:08:35.762217 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15496b43b694421d8783c489bd40ed58. Latest consensus state: current_term: 1 leader_uuid: "15496b43b694421d8783c489bd40ed58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15496b43b694421d8783c489bd40ed58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37729 } } }
I20260928 09:08:35.762456 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:35.762363 21554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15496b43b694421d8783c489bd40ed58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15496b43b694421d8783c489bd40ed58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37729 } } }
I20260928 09:08:35.762638 21554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:35.762959 21561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:35.763448 21561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:35.765077 21561 catalog_manager.cc:1347] Generated new cluster ID: cc95f0cf2b034825bbda1fde72b89947
I20260928 09:08:35.765128 21561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:35.785823 21561 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:35.786031 21561 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:35.795703 21561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15496b43b694421d8783c489bd40ed58: Generated new TSK 0
I20260928 09:08:35.795933 21561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:35.802515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21474
2026-09-28T09:08:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:35 dist-test-slave-smns krb5kdc[21573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:35 dist-test-slave-smns krb5kdc[21573](info): set up 2 sockets
Sep 28 09:08:35 dist-test-slave-smns krb5kdc[21573](info): commencing operation
krb5kdc: starting...
W20260928 09:08:36.829952 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:36 dist-test-slave-smns krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:36.953828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:37.037206 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:37.037462 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:37.037505 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:37.038827 21586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:37.038898 21586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:37.038924 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:37.038959 21586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:37.038990 21586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:37.040516 21586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:37.040797 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:37.041034 21586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:37.046380 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:37.046398 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:37.046574 21586 server_base.cc:1034] running on GCE node
W20260928 09:08:37.046398 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:37.046972 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:37.047309 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:37.048506 21586 hybrid_clock.cc:648] HybridClock initialized: now 1790586517048482 us; error 41 us; skew 500 ppm
Sep 28 09:08:37 dist-test-slave-smns krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:37.057456 21586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:37.058036 21586 webserver.cc:460] Webserver started at http://127.0.0.1:34639/ using document root <none> and password file <none>
I20260928 09:08:37.058282 21586 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:37.058332 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:37.058427 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:37.059484 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "5d4808286e0c4c4ea97b8cdc34983d0d"
format_stamp: "Formatted at 2026-09-28 09:08:37 on dist-test-slave-smns"
server_key: "6df8f002babaae8be794d125feea622b"
server_key_iv: "3272ba48573495c246e22b993c1b9dce"
server_key_version: "encryptionkey@0"
I20260928 09:08:37.059794 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d4808286e0c4c4ea97b8cdc34983d0d"
format_stamp: "Formatted at 2026-09-28 09:08:37 on dist-test-slave-smns"
server_key: "6df8f002babaae8be794d125feea622b"
server_key_iv: "3272ba48573495c246e22b993c1b9dce"
server_key_version: "encryptionkey@0"
I20260928 09:08:37.061152 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:37.062032 21601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:37.062239 21586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:08:37.062341 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "5d4808286e0c4c4ea97b8cdc34983d0d"
format_stamp: "Formatted at 2026-09-28 09:08:37 on dist-test-slave-smns"
server_key: "6df8f002babaae8be794d125feea622b"
server_key_iv: "3272ba48573495c246e22b993c1b9dce"
server_key_version: "encryptionkey@0"
I20260928 09:08:37.062423 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:37.075503 21586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:37.078011 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:37.078186 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:37.082371 21586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37431
I20260928 09:08:37.082420 21653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37431 every 8 connection(s)
I20260928 09:08:37.082758 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:37.083596 21654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:37.085856 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d: Bootstrap starting.
I20260928 09:08:37.086369 21654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:37.086596 21654 log.cc:824] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:37.087261 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d: No bootstrap required, opened a new log
I20260928 09:08:37.088691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21586
I20260928 09:08:37.088822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:37.088994 392 external_mini_cluster.cc:1428] Setting key 47d2da28909084a1cdbefb0fd4c04801
I20260928 09:08:37.088855 21654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37431 } }
I20260928 09:08:37.089035 21654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:37.089095 21654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4808286e0c4c4ea97b8cdc34983d0d, State: Initialized, Role: FOLLOWER
I20260928 09:08:37.089246 21654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37431 } }
I20260928 09:08:37.089303 21654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:37.089381 21654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:37.089488 21654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:37.090090 21654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37431 } }
I20260928 09:08:37.090175 21654 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4808286e0c4c4ea97b8cdc34983d0d; no voters:
I20260928 09:08:37.090349 21654 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:37.090389 21659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:37.090575 21659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [term 1 LEADER]: Becoming Leader. State: Replica: 5d4808286e0c4c4ea97b8cdc34983d0d, State: Running, Role: LEADER
I20260928 09:08:37.090603 21654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:37.090726 21659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37431 } }
I20260928 09:08:37.091224 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4808286e0c4c4ea97b8cdc34983d0d. Latest consensus state: current_term: 1 leader_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37431 } } }
I20260928 09:08:37.091436 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:37.091953 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4808286e0c4c4ea97b8cdc34983d0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37431 } } }
I20260928 09:08:37.092101 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:37.093714 21668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:37.094233 21668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:37.096231 21668 catalog_manager.cc:1347] Generated new cluster ID: 2b87681b9a4e4e6387059f6623482152
I20260928 09:08:37.096278 21668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:08:37 dist-test-slave-smns krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:08:37.103694 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:37.089505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:57901 (local address 127.0.0.1:37431)
0928 09:08:37.089725 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:37.089736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:37.089767 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:08:37.090095 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:37.090100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:37.090169 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:37.090305 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:37.091250 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:37.093432 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:37.094769 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:37.095453 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:37.095542 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:37.098774 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:37.098778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:37.098779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:37.098791 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:37.101407 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:37.102006 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:37.102010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:37.102011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:37.102142 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:37.102445 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:37.102446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:37.102447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:37.102816 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:37.103002 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:37.103340 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 09:08:37.103457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":120,"threads_started":1}
I20260928 09:08:37.105206 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:37.105583 21668 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:37.106277 21668 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:37.117825 21668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d4808286e0c4c4ea97b8cdc34983d0d: Generated new TSK 0
I20260928 09:08:37.118064 21668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:37.120566 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:37.110135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:48359 (local address 127.0.0.1:37431)
0928 09:08:37.110177 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:37.110179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:37.113729 (+ 3550us) server_negotiation.cc:407] Connection header received
0928 09:08:37.113807 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:37.113808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:37.113824 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:37.113874 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:37.114407 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:37.114956 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:37.115770 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:37.115997 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:37.116088 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:37.117442 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:37.117442 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:37.117443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:37.117448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:37.119275 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:37.119751 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:37.119753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:37.119753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:37.119834 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:37.120051 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:37.120052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:37.120052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:37.120180 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:37.120315 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:37.120406 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 09:08:37.120473 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:08:37.122634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21586
W20260928 09:08:37.126585 21571 connection.cc:535] client connection to 127.0.0.1:37431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37431): Connection reset by peer (error 104)
2026-09-28T09:08:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:37 dist-test-slave-smns krb5kdc[21685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:37 dist-test-slave-smns krb5kdc[21685](info): set up 2 sockets
Sep 28 09:08:37 dist-test-slave-smns krb5kdc[21685](info): commencing operation
krb5kdc: starting...
W20260928 09:08:38.153795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:38 dist-test-slave-smns krb5kdc[21685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:38.276381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:38.356959 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:38.357172 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:38.357194 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:38.358696 21698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:38.358755 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:38.358778 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:38.358791 21698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:38.358804 21698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:38.360571 21698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:38.360857 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:38.361110 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:38.366160 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:38.366228 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:38.366169 21706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:38.366317 21698 server_base.cc:1034] running on GCE node
I20260928 09:08:38.366590 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:38.366930 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:38.368103 21698 hybrid_clock.cc:648] HybridClock initialized: now 1790586518368082 us; error 32 us; skew 500 ppm
Sep 28 09:08:38 dist-test-slave-smns krb5kdc[21685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:38.370779 21698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:38.371271 21698 webserver.cc:460] Webserver started at http://127.0.0.1:39331/ using document root <none> and password file <none>
I20260928 09:08:38.371501 21698 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:38.371569 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:38.371731 21698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:38.372887 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d"
format_stamp: "Formatted at 2026-09-28 09:08:38 on dist-test-slave-smns"
server_key: "a3ea705258d02105c167c9000f55fb5a"
server_key_iv: "b8d7b4ebb27bbef089932b0d40ce6703"
server_key_version: "encryptionkey@0"
I20260928 09:08:38.373251 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d"
format_stamp: "Formatted at 2026-09-28 09:08:38 on dist-test-slave-smns"
server_key: "a3ea705258d02105c167c9000f55fb5a"
server_key_iv: "b8d7b4ebb27bbef089932b0d40ce6703"
server_key_version: "encryptionkey@0"
I20260928 09:08:38.374719 21698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:38.375496 21713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:38.375722 21698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:38.375824 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d"
format_stamp: "Formatted at 2026-09-28 09:08:38 on dist-test-slave-smns"
server_key: "a3ea705258d02105c167c9000f55fb5a"
server_key_iv: "b8d7b4ebb27bbef089932b0d40ce6703"
server_key_version: "encryptionkey@0"
I20260928 09:08:38.375913 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:38.389845 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:38.391837 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:38.392004 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:38.396019 21698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37527
I20260928 09:08:38.396068 21765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37527 every 8 connection(s)
I20260928 09:08:38.396447 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:38.397430 21766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:38.399736 21766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d: Bootstrap starting.
I20260928 09:08:38.400283 21766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:38.400513 21766 log.cc:824] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:38.400961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21698
I20260928 09:08:38.401095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:38.401275 392 external_mini_cluster.cc:1428] Setting key 89c05a7872fa0b2feb4de32a257fd170
I20260928 09:08:38.401270 21766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d: No bootstrap required, opened a new log
I20260928 09:08:38.404217 21766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37527 } }
I20260928 09:08:38.404383 21766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:38.404412 21766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6f2733a7684fe99b6f1d96ecd74b7d, State: Initialized, Role: FOLLOWER
I20260928 09:08:38.404529 21766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37527 } }
I20260928 09:08:38.404611 21766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:38.404650 21766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:38.404707 21766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:38.405273 21766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37527 } }
I20260928 09:08:38.405360 21766 leader_election.cc:302] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a6f2733a7684fe99b6f1d96ecd74b7d; no voters:
I20260928 09:08:38.405568 21766 leader_election.cc:288] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:38.405602 21771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:38.405777 21771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [term 1 LEADER]: Becoming Leader. State: Replica: 2a6f2733a7684fe99b6f1d96ecd74b7d, State: Running, Role: LEADER
I20260928 09:08:38.405822 21766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:38.405920 21771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37527 } }
I20260928 09:08:38.406262 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6f2733a7684fe99b6f1d96ecd74b7d. Latest consensus state: current_term: 1 leader_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37527 } } }
I20260928 09:08:38.406275 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6f2733a7684fe99b6f1d96ecd74b7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37527 } } }
I20260928 09:08:38.406383 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:38.406451 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:38.406628 21778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:38.407157 21778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:38.408216 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:38.401814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:38713 (local address 127.0.0.1:37527)
0928 09:08:38.402009 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:38.402019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:38.402056 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:08:38.402730 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:38.402740 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:38.402807 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:38.402980 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:38.403141 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:38.404585 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:38.405513 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:38.406192 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:38.407222 (+ 1030us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:38.407394 (+ 172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:38.407397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:38.407398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:38.407666 (+ 268us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:38.407695 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:38.407706 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:38.407805 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:38.407891 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 09:08:38.408011 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260928 09:08:38.409222 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:38.410597 21778 catalog_manager.cc:1347] Generated new cluster ID: 0ef681bb45de4bf0b21f40377cb88c7a
I20260928 09:08:38.410706 21778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:38.416682 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:38.410745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:44725 (local address 127.0.0.1:37527)
0928 09:08:38.410804 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:38.410806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:38.413783 (+ 2977us) server_negotiation.cc:407] Connection header received
0928 09:08:38.413855 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:38.413855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:38.413866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:38.413948 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:38.414194 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:38.414662 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:38.415425 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:38.415667 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:38.415760 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:38.416437 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:38.416439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:38.416439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:38.416444 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:38.416458 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:38.416462 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:38.416520 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:38.416591 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 09:08:38.416603 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:08:38.438252 21778 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:38.438877 21778 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:38.442735 21778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a6f2733a7684fe99b6f1d96ecd74b7d: Generated new TSK 0
I20260928 09:08:38.442955 21778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:38.447300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21698
W20260928 09:08:38.451588 21683 connection.cc:535] client connection to 127.0.0.1:37527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37527): Connection reset by peer (error 104)
2026-09-28T09:08:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:38 dist-test-slave-smns krb5kdc[21797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:38 dist-test-slave-smns krb5kdc[21797](info): set up 2 sockets
Sep 28 09:08:38 dist-test-slave-smns krb5kdc[21797](info): commencing operation
krb5kdc: starting...
W20260928 09:08:39.505759 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:39 dist-test-slave-smns krb5kdc[21797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:08:39.628373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:39.711944 21810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:39.712150 21810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:39.712173 21810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:39.713423 21810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:39.713465 21810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:39.713479 21810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:39.713492 21810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:39.713505 21810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:39.715054 21810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:39.715328 21810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:39.715565 21810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:39.720755 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:39.720762 21815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:39.720899 21810 server_base.cc:1034] running on GCE node
W20260928 09:08:39.720762 21816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:39.721395 21810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:39.721774 21810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:39.722941 21810 hybrid_clock.cc:648] HybridClock initialized: now 1790586519722938 us; error 36 us; skew 500 ppm
Sep 28 09:08:39 dist-test-slave-smns krb5kdc[21797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:39.725698 21810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:08:39.726194 21810 webserver.cc:460] Webserver started at http://127.0.0.1:42559/ using document root <none> and password file <none>
I20260928 09:08:39.726446 21810 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:39.726516 21810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:39.726650 21810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:39.727736 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "95e70e8f8a2846f9898300e927571526"
format_stamp: "Formatted at 2026-09-28 09:08:39 on dist-test-slave-smns"
server_key: "82dd7783ab170c871c4825b661b6d583"
server_key_iv: "21a7ffa3227f2a6d46145fc1f191f3ff"
server_key_version: "encryptionkey@0"
I20260928 09:08:39.728109 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "95e70e8f8a2846f9898300e927571526"
format_stamp: "Formatted at 2026-09-28 09:08:39 on dist-test-slave-smns"
server_key: "82dd7783ab170c871c4825b661b6d583"
server_key_iv: "21a7ffa3227f2a6d46145fc1f191f3ff"
server_key_version: "encryptionkey@0"
I20260928 09:08:39.729497 21810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:39.730372 21825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:39.730561 21810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:39.730662 21810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "95e70e8f8a2846f9898300e927571526"
format_stamp: "Formatted at 2026-09-28 09:08:39 on dist-test-slave-smns"
server_key: "82dd7783ab170c871c4825b661b6d583"
server_key_iv: "21a7ffa3227f2a6d46145fc1f191f3ff"
server_key_version: "encryptionkey@0"
I20260928 09:08:39.730754 21810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:39.734236 21810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:39.736120 21810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:39.736284 21810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:39.740252 21810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46319
I20260928 09:08:39.740279 21877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46319 every 8 connection(s)
I20260928 09:08:39.740597 21810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:39.741611 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:39.743156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21810
I20260928 09:08:39.743280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:39.743463 392 external_mini_cluster.cc:1428] Setting key a8f75da9813d26ad36620f9c4b9cffa9
I20260928 09:08:39.744069 21878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526: Bootstrap starting.
I20260928 09:08:39.744634 21878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:39.744988 21878 log.cc:824] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:39.745800 21878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526: No bootstrap required, opened a new log
I20260928 09:08:39.746075 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:39.743988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:40375 (local address 127.0.0.1:46319)
0928 09:08:39.744171 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:39.744181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:39.744262 (+ 81us) server_negotiation.cc:407] Connection header received
0928 09:08:39.744667 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:39.744672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:39.744695 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:39.744855 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:39.745510 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:39.745569 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:39.745570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:39.745575 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:39.745695 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:39.745709 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:39.745804 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:39.745859 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 09:08:39.745901 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 09:08:39.747035 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:39.748119 21878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e70e8f8a2846f9898300e927571526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260928 09:08:39.748311 21878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:39.748377 21878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e70e8f8a2846f9898300e927571526, State: Initialized, Role: FOLLOWER
I20260928 09:08:39.748508 21878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e70e8f8a2846f9898300e927571526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260928 09:08:39.748611 21878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:39.748647 21878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:39.748700 21878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:39.749425 21878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e70e8f8a2846f9898300e927571526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260928 09:08:39.749534 21878 leader_election.cc:302] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95e70e8f8a2846f9898300e927571526; no voters:
I20260928 09:08:39.749711 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:39.748752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:58683 (local address 127.0.0.1:46319)
0928 09:08:39.748792 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:39.748794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:39.748806 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:08:39.748890 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:39.748891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:39.748894 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:39.748938 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:39.749449 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:39.749450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:39.749450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:39.749453 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:39.749464 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:39.749467 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:39.749542 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:39.749621 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 09:08:39.749635 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:08:39.749714 21878 leader_election.cc:288] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:39.749756 21888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:39.750015 21878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:39.750039 21888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [term 1 LEADER]: Becoming Leader. State: Replica: 95e70e8f8a2846f9898300e927571526, State: Running, Role: LEADER
I20260928 09:08:39.750293 21888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e70e8f8a2846f9898300e927571526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260928 09:08:39.750717 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e70e8f8a2846f9898300e927571526. Latest consensus state: current_term: 1 leader_uuid: "95e70e8f8a2846f9898300e927571526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e70e8f8a2846f9898300e927571526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } } }
I20260928 09:08:39.750810 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:39.751040 21897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:39.751202 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95e70e8f8a2846f9898300e927571526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e70e8f8a2846f9898300e927571526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } } }
I20260928 09:08:39.751334 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:39.751611 21897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:39.753115 21897 catalog_manager.cc:1347] Generated new cluster ID: 9d02cac3c4774184a0711a1cc7d873c2
I20260928 09:08:39.753165 21897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:39.777383 21897 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:39.777601 21897 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:39.780817 21897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95e70e8f8a2846f9898300e927571526: Generated new TSK 0
I20260928 09:08:39.781006 21897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:39.786816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21810
2026-09-28T09:08:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:39 dist-test-slave-smns krb5kdc[21909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:39 dist-test-slave-smns krb5kdc[21909](info): set up 2 sockets
Sep 28 09:08:39 dist-test-slave-smns krb5kdc[21909](info): commencing operation
krb5kdc: starting...
W20260928 09:08:40.825794 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:40 dist-test-slave-smns krb5kdc[21909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:40.949280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:41.046569 21922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:41.046779 21922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:41.046802 21922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:41.047991 21922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:41.048036 21922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:41.048050 21922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:41.048063 21922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:41.048105 21922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:41.049672 21922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:41.049932 21922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:41.050190 21922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:41.055116 21930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:41.055157 21922 server_base.cc:1034] running on GCE node
W20260928 09:08:41.055123 21927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:41.055133 21928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:41.055583 21922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:41.055908 21922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:41.057093 21922 hybrid_clock.cc:648] HybridClock initialized: now 1790586521057077 us; error 46 us; skew 500 ppm
Sep 28 09:08:41 dist-test-slave-smns krb5kdc[21909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:41.060088 21922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:41.060765 21922 webserver.cc:460] Webserver started at http://127.0.98.62:33385/ using document root <none> and password file <none>
I20260928 09:08:41.061074 21922 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:41.061170 21922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:41.061287 21922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:41.062541 21922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "6b8064c3e49d441dbabe44e9cff6dfc2"
format_stamp: "Formatted at 2026-09-28 09:08:41 on dist-test-slave-smns"
server_key: "38feaff8dd4bb201d0930115c32d321c"
server_key_iv: "1915042095a7745acb115da29608fd39"
server_key_version: "encryptionkey@0"
I20260928 09:08:41.062871 21922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b8064c3e49d441dbabe44e9cff6dfc2"
format_stamp: "Formatted at 2026-09-28 09:08:41 on dist-test-slave-smns"
server_key: "38feaff8dd4bb201d0930115c32d321c"
server_key_iv: "1915042095a7745acb115da29608fd39"
server_key_version: "encryptionkey@0"
I20260928 09:08:41.064419 21922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:41.065215 21937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:41.065414 21922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:41.065536 21922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "6b8064c3e49d441dbabe44e9cff6dfc2"
format_stamp: "Formatted at 2026-09-28 09:08:41 on dist-test-slave-smns"
server_key: "38feaff8dd4bb201d0930115c32d321c"
server_key_iv: "1915042095a7745acb115da29608fd39"
server_key_version: "encryptionkey@0"
I20260928 09:08:41.065703 21922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:41.088105 21922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:41.090166 21922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:41.090343 21922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:41.094383 21922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36683
I20260928 09:08:41.094455 21989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36683 every 8 connection(s)
I20260928 09:08:41.094817 21922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:41.095742 21990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:41.098286 21990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2: Bootstrap starting.
I20260928 09:08:41.098856 21990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:41.099107 21990 log.cc:824] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:41.099818 21990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2: No bootstrap required, opened a new log
I20260928 09:08:41.101529 21990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 09:08:41.101718 21990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:41.101743 21990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8064c3e49d441dbabe44e9cff6dfc2, State: Initialized, Role: FOLLOWER
I20260928 09:08:41.101900 21990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 09:08:41.101989 21990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:41.102015 21990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:41.102046 21990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:41.102576 21990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 09:08:41.102677 21990 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b8064c3e49d441dbabe44e9cff6dfc2; no voters:
I20260928 09:08:41.102806 21990 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:41.102883 21993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:41.103047 21993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 6b8064c3e49d441dbabe44e9cff6dfc2, State: Running, Role: LEADER
I20260928 09:08:41.103112 21990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:41.103168 21993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 09:08:41.103595 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8064c3e49d441dbabe44e9cff6dfc2. Latest consensus state: current_term: 1 leader_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } } }
I20260928 09:08:41.103744 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:41.103717 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8064c3e49d441dbabe44e9cff6dfc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } } }
I20260928 09:08:41.103853 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:41.104243 22002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:41.104445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 21922
I20260928 09:08:41.104570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:41.104779 392 external_mini_cluster.cc:1428] Setting key 12d485d2f761982bfab92b3fe9071836
I20260928 09:08:41.104923 22002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:41.106711 22002 catalog_manager.cc:1347] Generated new cluster ID: f9ac8727558942338342735dcbbbbfa4
I20260928 09:08:41.106768 22002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:08:41 dist-test-slave-smns krb5kdc[21909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:41.115885 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:41.105477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60893 (local address 127.0.98.62:36683)
0928 09:08:41.105702 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:41.105711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:41.105742 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:08:41.105855 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:41.105862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:41.105916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:41.106178 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:41.107013 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:41.108039 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:41.108923 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:41.109377 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:41.109531 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:41.112146 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:41.112150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:41.112150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:41.112164 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:41.114225 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:41.114655 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:41.114659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:41.114659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:41.114772 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:41.115009 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:41.115010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:41.115010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:41.115272 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:41.115461 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:41.115548 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 09:08:41.115653 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
I20260928 09:08:41.116882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:41.124269 22002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:41.125031 22002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:41.128969 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:41.122016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54263 (local address 127.0.98.62:36683)
0928 09:08:41.122064 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:41.122066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:41.122222 (+ 156us) server_negotiation.cc:407] Connection header received
0928 09:08:41.122299 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:41.122300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:41.122311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:41.122407 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:41.123006 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:41.123576 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:41.124493 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:41.124721 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:41.124801 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:41.126183 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:41.126185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:41.126186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:41.126193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:41.127745 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:41.128130 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:41.128131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:41.128131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:41.128194 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:41.128436 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:41.128437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:41.128438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:41.128548 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:41.128675 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:41.128787 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 09:08:41.128861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:08:41.139340 22002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8064c3e49d441dbabe44e9cff6dfc2: Generated new TSK 0
I20260928 09:08:41.139544 22002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:41.145749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 21922
W20260928 09:08:41.150111 21907 connection.cc:535] client connection to 127.0.98.62:36683 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36683): Connection reset by peer (error 104)
2026-09-28T09:08:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:41 dist-test-slave-smns krb5kdc[22021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:41 dist-test-slave-smns krb5kdc[22021](info): set up 2 sockets
Sep 28 09:08:41 dist-test-slave-smns krb5kdc[22021](info): commencing operation
krb5kdc: starting...
W20260928 09:08:42.193647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:42 dist-test-slave-smns krb5kdc[22021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:42.317771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39267
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:42.402163 22034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:42.402401 22034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:42.402431 22034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:42.403671 22034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:42.403716 22034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:42.403731 22034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:42.403744 22034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:42.403756 22034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:42.405344 22034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:42.405654 22034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:42.405867 22034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:42.410995 22039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:42.410995 22040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:42.411368 22042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:42.411532 22034 server_base.cc:1034] running on GCE node
I20260928 09:08:42.411795 22034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:42.412118 22034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:42.413295 22034 hybrid_clock.cc:648] HybridClock initialized: now 1790586522413277 us; error 31 us; skew 500 ppm
Sep 28 09:08:42 dist-test-slave-smns krb5kdc[22021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:42.425838 22034 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:42.426313 22034 webserver.cc:460] Webserver started at http://127.0.98.62:43117/ using document root <none> and password file <none>
I20260928 09:08:42.426577 22034 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:42.426626 22034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:42.426769 22034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:42.427950 22034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "0fc89b96b1684749b46be067a34e449a"
format_stamp: "Formatted at 2026-09-28 09:08:42 on dist-test-slave-smns"
server_key: "c0cb623af75556bac8b29567887bba7d"
server_key_iv: "62b284cde28024943b7b57f6443c9deb"
server_key_version: "encryptionkey@0"
I20260928 09:08:42.428318 22034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fc89b96b1684749b46be067a34e449a"
format_stamp: "Formatted at 2026-09-28 09:08:42 on dist-test-slave-smns"
server_key: "c0cb623af75556bac8b29567887bba7d"
server_key_iv: "62b284cde28024943b7b57f6443c9deb"
server_key_version: "encryptionkey@0"
I20260928 09:08:42.429840 22034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:42.430828 22049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:42.431134 22034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:42.431244 22034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "0fc89b96b1684749b46be067a34e449a"
format_stamp: "Formatted at 2026-09-28 09:08:42 on dist-test-slave-smns"
server_key: "c0cb623af75556bac8b29567887bba7d"
server_key_iv: "62b284cde28024943b7b57f6443c9deb"
server_key_version: "encryptionkey@0"
I20260928 09:08:42.431357 22034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:42.465435 22034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:42.468138 22034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:42.468305 22034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:42.472620 22034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39267
I20260928 09:08:42.472682 22101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39267 every 8 connection(s)
I20260928 09:08:42.473068 22034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:42.473322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22034
I20260928 09:08:42.473452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:42.473701 392 external_mini_cluster.cc:1428] Setting key eae14810dd7f7c90e298bf4da2519057
I20260928 09:08:42.474197 22102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:42.477569 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a: Bootstrap starting.
I20260928 09:08:42.478349 22102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:42.479074 22102 log.cc:824] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:42.479776 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:42.474265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54851 (local address 127.0.98.62:39267)
0928 09:08:42.474452 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:42.474460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:42.474488 (+ 28us) server_negotiation.cc:407] Connection header received
0928 09:08:42.474937 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:42.474943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:42.474990 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:42.475116 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:42.475521 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:42.476474 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:42.477643 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:42.478116 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:42.478208 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:42.479033 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:42.479037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:42.479038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:42.479264 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:42.479300 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:42.479312 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:42.479419 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:42.479509 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 09:08:42.479542 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260928 09:08:42.479810 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a: No bootstrap required, opened a new log
I20260928 09:08:42.480989 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:42.482152 22102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc89b96b1684749b46be067a34e449a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39267 } }
I20260928 09:08:42.482302 22102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:42.482359 22102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc89b96b1684749b46be067a34e449a, State: Initialized, Role: FOLLOWER
I20260928 09:08:42.482550 22102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc89b96b1684749b46be067a34e449a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39267 } }
I20260928 09:08:42.482646 22102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:42.482700 22102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:42.482752 22102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:42.483366 22102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc89b96b1684749b46be067a34e449a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39267 } }
I20260928 09:08:42.483490 22102 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc89b96b1684749b46be067a34e449a; no voters:
I20260928 09:08:42.483698 22102 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:42.483759 22112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:42.483927 22112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [term 1 LEADER]: Becoming Leader. State: Replica: 0fc89b96b1684749b46be067a34e449a, State: Running, Role: LEADER
I20260928 09:08:42.483960 22102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:42.484083 22112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc89b96b1684749b46be067a34e449a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39267 } }
I20260928 09:08:42.484422 22112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc89b96b1684749b46be067a34e449a. Latest consensus state: current_term: 1 leader_uuid: "0fc89b96b1684749b46be067a34e449a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc89b96b1684749b46be067a34e449a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39267 } } }
I20260928 09:08:42.484432 22113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc89b96b1684749b46be067a34e449a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc89b96b1684749b46be067a34e449a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39267 } } }
I20260928 09:08:42.484606 22113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:42.484575 22112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:42.484849 22121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:42.485395 22121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:42.486263 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:42.482446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47049 (local address 127.0.98.62:39267)
0928 09:08:42.482489 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:42.482491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:42.482619 (+ 128us) server_negotiation.cc:407] Connection header received
0928 09:08:42.482746 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:42.482747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:42.482759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:42.482811 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:42.482976 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:42.483466 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:42.484665 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:42.484979 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:42.485033 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:42.485975 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:42.485976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:42.485977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:42.485981 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:42.485997 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:42.486002 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:42.486070 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:42.486163 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 09:08:42.486182 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:08:42.487452 22121 catalog_manager.cc:1347] Generated new cluster ID: 3e01ebc3b6004004bf631b593e5a81dc
I20260928 09:08:42.487504 22121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:42.501748 22121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:42.502326 22121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:42.506716 22121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc89b96b1684749b46be067a34e449a: Generated new TSK 0
I20260928 09:08:42.506918 22121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:42.514555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22034
W20260928 09:08:42.519160 22019 connection.cc:535] client connection to 127.0.98.62:39267 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39267): Connection reset by peer (error 104)
2026-09-28T09:08:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:42 dist-test-slave-smns krb5kdc[22133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:42 dist-test-slave-smns krb5kdc[22133](info): set up 2 sockets
Sep 28 09:08:42 dist-test-slave-smns krb5kdc[22133](info): commencing operation
krb5kdc: starting...
W20260928 09:08:43.561941 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:43 dist-test-slave-smns krb5kdc[22133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:43.689457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:43.772677 22146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:43.772930 22146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:43.772972 22146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:43.774266 22146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:43.774338 22146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:43.774362 22146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:43.774384 22146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:43.774411 22146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:43.775933 22146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:43.776226 22146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:43.776469 22146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:43.781760 22152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:43.781776 22151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:43.781775 22154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:43.781804 22146 server_base.cc:1034] running on GCE node
I20260928 09:08:43.782229 22146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:43.782559 22146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:43.783761 22146 hybrid_clock.cc:648] HybridClock initialized: now 1790586523783754 us; error 37 us; skew 500 ppm
Sep 28 09:08:43 dist-test-slave-smns krb5kdc[22133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:43.786414 22146 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:43.786955 22146 webserver.cc:460] Webserver started at http://127.0.98.62:40253/ using document root <none> and password file <none>
I20260928 09:08:43.787221 22146 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:43.787294 22146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:43.787441 22146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:43.788509 22146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "74c761ed33dc48a885a62134cfd592e1"
format_stamp: "Formatted at 2026-09-28 09:08:43 on dist-test-slave-smns"
server_key: "62957859a0084eb313ecfd9ca6ce8a7d"
server_key_iv: "add8fcb783f4f7818f106ebfa5e7f779"
server_key_version: "encryptionkey@0"
I20260928 09:08:43.788945 22146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "74c761ed33dc48a885a62134cfd592e1"
format_stamp: "Formatted at 2026-09-28 09:08:43 on dist-test-slave-smns"
server_key: "62957859a0084eb313ecfd9ca6ce8a7d"
server_key_iv: "add8fcb783f4f7818f106ebfa5e7f779"
server_key_version: "encryptionkey@0"
I20260928 09:08:43.790416 22146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:08:43.791219 22161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:43.791452 22146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:43.791566 22146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "74c761ed33dc48a885a62134cfd592e1"
format_stamp: "Formatted at 2026-09-28 09:08:43 on dist-test-slave-smns"
server_key: "62957859a0084eb313ecfd9ca6ce8a7d"
server_key_iv: "add8fcb783f4f7818f106ebfa5e7f779"
server_key_version: "encryptionkey@0"
I20260928 09:08:43.791644 22146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:43.795758 22146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:43.797842 22146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:43.797999 22146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:43.802218 22146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35205
I20260928 09:08:43.802266 22213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35205 every 8 connection(s)
I20260928 09:08:43.802629 22146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:43.803579 22214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:43.804049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22146
I20260928 09:08:43.804167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:43.804337 392 external_mini_cluster.cc:1428] Setting key 48bf52738a22649939c6d7b68ce4a057
I20260928 09:08:43.806533 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1: Bootstrap starting.
I20260928 09:08:43.807035 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:43.804915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34699 (local address 127.0.98.62:35205)
0928 09:08:43.805105 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:43.805113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:43.805139 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:08:43.805605 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:43.805613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:43.805646 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:43.805789 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:43.806529 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:43.806533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:43.806533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:43.806538 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:43.806606 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:43.806621 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:43.806688 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:43.806765 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 09:08:43.806895 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260928 09:08:43.807188 22214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:43.807487 22214 log.cc:824] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:43.808166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:43.808357 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1: No bootstrap required, opened a new log
I20260928 09:08:43.810649 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:43.809536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33679 (local address 127.0.98.62:35205)
0928 09:08:43.809592 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:43.809594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:43.809720 (+ 126us) server_negotiation.cc:407] Connection header received
0928 09:08:43.809783 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:43.809784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:43.809800 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:43.809861 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:43.810442 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:43.810443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:43.810444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:43.810448 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:43.810468 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:43.810474 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:43.810540 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:43.810582 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 09:08:43.810594 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:08:43.810833 22214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c761ed33dc48a885a62134cfd592e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35205 } }
I20260928 09:08:43.811018 22214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:43.811095 22214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c761ed33dc48a885a62134cfd592e1, State: Initialized, Role: FOLLOWER
I20260928 09:08:43.811244 22214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c761ed33dc48a885a62134cfd592e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35205 } }
I20260928 09:08:43.811329 22214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:43.811384 22214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:43.811439 22214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:43.812036 22214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c761ed33dc48a885a62134cfd592e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35205 } }
I20260928 09:08:43.812127 22214 leader_election.cc:302] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74c761ed33dc48a885a62134cfd592e1; no voters:
I20260928 09:08:43.812316 22214 leader_election.cc:288] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:43.812414 22224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:43.812557 22214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:43.812559 22224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [term 1 LEADER]: Becoming Leader. State: Replica: 74c761ed33dc48a885a62134cfd592e1, State: Running, Role: LEADER
I20260928 09:08:43.812678 22224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c761ed33dc48a885a62134cfd592e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35205 } }
I20260928 09:08:43.813063 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c761ed33dc48a885a62134cfd592e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c761ed33dc48a885a62134cfd592e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35205 } } }
I20260928 09:08:43.813077 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c761ed33dc48a885a62134cfd592e1. Latest consensus state: current_term: 1 leader_uuid: "74c761ed33dc48a885a62134cfd592e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c761ed33dc48a885a62134cfd592e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35205 } } }
I20260928 09:08:43.813314 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:43.813318 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:43.813704 22231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:43.814200 22231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:43.815711 22231 catalog_manager.cc:1347] Generated new cluster ID: 2527f1497f1d4757abb1efc311804035
I20260928 09:08:43.815769 22231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:43.825510 22231 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:43.825721 22231 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:43.834671 22231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c761ed33dc48a885a62134cfd592e1: Generated new TSK 0
I20260928 09:08:43.834923 22231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:43.840405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22146
2026-09-28T09:08:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:43 dist-test-slave-smns krb5kdc[22245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:43 dist-test-slave-smns krb5kdc[22245](info): set up 2 sockets
Sep 28 09:08:43 dist-test-slave-smns krb5kdc[22245](info): commencing operation
krb5kdc: starting...
W20260928 09:08:44.889827 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:44 dist-test-slave-smns krb5kdc[22245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:45.012893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:45.095867 22258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:45.096114 22258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:45.096153 22258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:45.097314 22258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:45.097384 22258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:45.097409 22258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:45.097431 22258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:45.097460 22258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:45.098971 22258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:45.099251 22258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:45.099481 22258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:45.104439 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:45.104435 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:45.104714 22258 server_base.cc:1034] running on GCE node
W20260928 09:08:45.104496 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:45.105016 22258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:45.105298 22258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:45.106460 22258 hybrid_clock.cc:648] HybridClock initialized: now 1790586525106428 us; error 49 us; skew 500 ppm
Sep 28 09:08:45 dist-test-slave-smns krb5kdc[22245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:45.109150 22258 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:45.109644 22258 webserver.cc:460] Webserver started at http://127.0.98.62:40819/ using document root <none> and password file <none>
I20260928 09:08:45.109874 22258 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:45.109941 22258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:45.110078 22258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:45.111176 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "7d3f644b01304634ab05787a4aba0eef"
format_stamp: "Formatted at 2026-09-28 09:08:45 on dist-test-slave-smns"
server_key: "2d91bf26790c696caca8558125092014"
server_key_iv: "0dbc7dc00df07aa5b4f18ea44881926c"
server_key_version: "encryptionkey@0"
I20260928 09:08:45.111634 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d3f644b01304634ab05787a4aba0eef"
format_stamp: "Formatted at 2026-09-28 09:08:45 on dist-test-slave-smns"
server_key: "2d91bf26790c696caca8558125092014"
server_key_iv: "0dbc7dc00df07aa5b4f18ea44881926c"
server_key_version: "encryptionkey@0"
I20260928 09:08:45.113054 22258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:45.113924 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:45.114084 22258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:45.114185 22258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "7d3f644b01304634ab05787a4aba0eef"
format_stamp: "Formatted at 2026-09-28 09:08:45 on dist-test-slave-smns"
server_key: "2d91bf26790c696caca8558125092014"
server_key_iv: "0dbc7dc00df07aa5b4f18ea44881926c"
server_key_version: "encryptionkey@0"
I20260928 09:08:45.114348 22258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:45.122175 22258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:45.124486 22258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:45.124682 22258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:45.128850 22258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43239
I20260928 09:08:45.128890 22325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43239 every 8 connection(s)
I20260928 09:08:45.129258 22258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:45.130195 22326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:45.132431 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef: Bootstrap starting.
I20260928 09:08:45.132969 22326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:45.133195 22326 log.cc:824] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:45.133922 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef: No bootstrap required, opened a new log
I20260928 09:08:45.135485 22326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3f644b01304634ab05787a4aba0eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260928 09:08:45.135628 22326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:45.135650 22326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3f644b01304634ab05787a4aba0eef, State: Initialized, Role: FOLLOWER
I20260928 09:08:45.135790 22326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3f644b01304634ab05787a4aba0eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260928 09:08:45.135874 22326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:45.135900 22326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:45.135927 22326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:45.136408 22326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3f644b01304634ab05787a4aba0eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260928 09:08:45.136492 22326 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3f644b01304634ab05787a4aba0eef; no voters:
I20260928 09:08:45.136606 22326 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:45.136672 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:45.136844 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [term 1 LEADER]: Becoming Leader. State: Replica: 7d3f644b01304634ab05787a4aba0eef, State: Running, Role: LEADER
I20260928 09:08:45.136902 22326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:45.136950 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3f644b01304634ab05787a4aba0eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260928 09:08:45.137601 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3f644b01304634ab05787a4aba0eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3f644b01304634ab05787a4aba0eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } } }
I20260928 09:08:45.137693 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:45.137744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22258
I20260928 09:08:45.137858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:45.138043 392 external_mini_cluster.cc:1428] Setting key 07bb950c5326434686827fab0f230a3e
I20260928 09:08:45.138320 22338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:45.138283 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3f644b01304634ab05787a4aba0eef. Latest consensus state: current_term: 1 leader_uuid: "7d3f644b01304634ab05787a4aba0eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3f644b01304634ab05787a4aba0eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } } }
I20260928 09:08:45.138801 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:45.139228 22338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:45.140695 22338 catalog_manager.cc:1347] Generated new cluster ID: ff87780ab4f94396ae3a2996dd427a3b
I20260928 09:08:45.140750 22338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:08:45 dist-test-slave-smns krb5kdc[22245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:45.147953 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:45.138584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47895 (local address 127.0.98.62:43239)
0928 09:08:45.138764 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:45.138774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:45.138851 (+ 77us) server_negotiation.cc:407] Connection header received
0928 09:08:45.138975 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:45.138980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:45.139025 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:45.139176 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:45.139829 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:45.140943 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:45.141821 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:45.142232 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:45.142356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:45.144665 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:45.144669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:45.144669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:45.144682 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:45.146428 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:45.146811 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:45.146814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:45.146814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:45.146924 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:45.147151 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:45.147151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:45.147152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:45.147388 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:45.147551 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:45.147623 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 09:08:45.147765 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 09:08:45.148874 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:45.160179 22338 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:45.160686 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:45.153874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56255 (local address 127.0.98.62:43239)
0928 09:08:45.153915 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:45.153917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:45.154077 (+ 160us) server_negotiation.cc:407] Connection header received
0928 09:08:45.154142 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:45.154143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:45.154153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:45.154212 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:45.154799 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:45.155351 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:45.156100 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:45.156331 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:45.156420 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:45.157775 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:45.157776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:45.157777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:45.157784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:45.159308 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:45.159748 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:45.159749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:45.159749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:45.159809 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:45.160021 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:45.160022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:45.160022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:45.160172 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:45.160279 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:45.160500 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 09:08:45.160587 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:08:45.160938 22338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:45.166070 22338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3f644b01304634ab05787a4aba0eef: Generated new TSK 0
I20260928 09:08:45.166298 22338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:45.172222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22258
2026-09-28T09:08:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:45 dist-test-slave-smns krb5kdc[22357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:45 dist-test-slave-smns krb5kdc[22357](info): set up 2 sockets
Sep 28 09:08:45 dist-test-slave-smns krb5kdc[22357](info): commencing operation
krb5kdc: starting...
W20260928 09:08:46.205839 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:46 dist-test-slave-smns krb5kdc[22357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:46.328619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34809
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:46.410480 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:46.410732 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:46.410771 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:46.411960 22370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:46.412031 22370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:46.412055 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:46.412084 22370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:46.412117 22370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:46.413602 22370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:46.413878 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:46.414103 22370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:46.419267 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:46.419255 22378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:46.419313 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:46.419467 22370 server_base.cc:1034] running on GCE node
I20260928 09:08:46.419762 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:46.420025 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:46.421164 22370 hybrid_clock.cc:648] HybridClock initialized: now 1790586526421112 us; error 63 us; skew 500 ppm
Sep 28 09:08:46 dist-test-slave-smns krb5kdc[22357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:46.423756 22370 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:46.424221 22370 webserver.cc:460] Webserver started at http://127.0.98.62:35089/ using document root <none> and password file <none>
I20260928 09:08:46.424470 22370 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:46.424541 22370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:46.424693 22370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:46.425871 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "7db63272895d4b429b83fa8ebd200978"
format_stamp: "Formatted at 2026-09-28 09:08:46 on dist-test-slave-smns"
server_key: "1749bf53175e0268a3fcb5d97d3a3102"
server_key_iv: "02032ba7f37bde8d6c5928fd8368d433"
server_key_version: "encryptionkey@0"
I20260928 09:08:46.426227 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "7db63272895d4b429b83fa8ebd200978"
format_stamp: "Formatted at 2026-09-28 09:08:46 on dist-test-slave-smns"
server_key: "1749bf53175e0268a3fcb5d97d3a3102"
server_key_iv: "02032ba7f37bde8d6c5928fd8368d433"
server_key_version: "encryptionkey@0"
I20260928 09:08:46.427594 22370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:46.428442 22385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:46.428629 22370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:46.428751 22370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "7db63272895d4b429b83fa8ebd200978"
format_stamp: "Formatted at 2026-09-28 09:08:46 on dist-test-slave-smns"
server_key: "1749bf53175e0268a3fcb5d97d3a3102"
server_key_iv: "02032ba7f37bde8d6c5928fd8368d433"
server_key_version: "encryptionkey@0"
I20260928 09:08:46.428828 22370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:46.440433 22370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:46.442876 22370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:46.443055 22370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:46.447367 22370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34809
I20260928 09:08:46.447410 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34809 every 8 connection(s)
I20260928 09:08:46.447806 22370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:46.448701 22438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:46.450953 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978: Bootstrap starting.
I20260928 09:08:46.451481 22438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:46.451711 22438 log.cc:824] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:46.452378 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978: No bootstrap required, opened a new log
I20260928 09:08:46.453514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22370
I20260928 09:08:46.453671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:46.453894 392 external_mini_cluster.cc:1428] Setting key 3d6395793d74284289d69ff357101b28
I20260928 09:08:46.454196 22438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63272895d4b429b83fa8ebd200978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34809 } }
I20260928 09:08:46.454452 22438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:46.454506 22438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db63272895d4b429b83fa8ebd200978, State: Initialized, Role: FOLLOWER
I20260928 09:08:46.454674 22438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63272895d4b429b83fa8ebd200978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34809 } }
I20260928 09:08:46.454767 22438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:46.454798 22438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:46.454852 22438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:46.455530 22438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63272895d4b429b83fa8ebd200978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34809 } }
I20260928 09:08:46.455619 22438 leader_election.cc:302] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db63272895d4b429b83fa8ebd200978; no voters:
I20260928 09:08:46.455793 22438 leader_election.cc:288] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:46.455832 22443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:46.455982 22443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [term 1 LEADER]: Becoming Leader. State: Replica: 7db63272895d4b429b83fa8ebd200978, State: Running, Role: LEADER
I20260928 09:08:46.456049 22438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:46.456120 22443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63272895d4b429b83fa8ebd200978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34809 } }
I20260928 09:08:46.456506 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db63272895d4b429b83fa8ebd200978. Latest consensus state: current_term: 1 leader_uuid: "7db63272895d4b429b83fa8ebd200978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63272895d4b429b83fa8ebd200978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34809 } } }
I20260928 09:08:46.456512 22445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db63272895d4b429b83fa8ebd200978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63272895d4b429b83fa8ebd200978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34809 } } }
I20260928 09:08:46.456670 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:46.456682 22445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:46.456918 22452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:46.457718 22452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:46.459458 22452 catalog_manager.cc:1347] Generated new cluster ID: 953b68d472374d558a1e76e2b6e3eefa
I20260928 09:08:46.459512 22452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:46.459928 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:46.454466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50317 (local address 127.0.98.62:34809)
0928 09:08:46.454683 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:46.454692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:46.454725 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:08:46.454862 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:46.454875 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:46.454945 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:46.455066 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:46.455362 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:46.456586 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:46.457827 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:46.458409 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:46.458517 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:46.459207 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:46.459211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:46.459211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:46.459462 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:46.459491 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:46.459518 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:46.459583 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:46.459694 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 09:08:46.459759 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260928 09:08:46.460805 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:46.468518 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:46.462221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54353 (local address 127.0.98.62:34809)
0928 09:08:46.462255 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:46.462257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:46.465730 (+ 3473us) server_negotiation.cc:407] Connection header received
0928 09:08:46.465804 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:46.465805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:46.465816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:46.465861 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:46.466045 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:46.466525 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:46.467277 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:46.467516 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:46.467596 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:46.468267 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:46.468269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:46.468269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:46.468273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:46.468288 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:46.468291 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:46.468344 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:46.468399 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 09:08:46.468423 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:08:46.499362 22452 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:46.499934 22452 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:46.503218 22452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db63272895d4b429b83fa8ebd200978: Generated new TSK 0
I20260928 09:08:46.503417 22452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:46.510511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22370
2026-09-28T09:08:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:46 dist-test-slave-smns krb5kdc[22469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:46 dist-test-slave-smns krb5kdc[22469](info): set up 2 sockets
Sep 28 09:08:46 dist-test-slave-smns krb5kdc[22469](info): commencing operation
krb5kdc: starting...
W20260928 09:08:47.545832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:47 dist-test-slave-smns krb5kdc[22469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:47.674111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37203
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:47.760412 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:47.760694 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:47.760741 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:47.762068 22482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:47.762151 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:47.762176 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:47.762199 22482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:47.762228 22482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:47.763916 22482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:47.764220 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:47.764573 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:47.769860 22488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:47.769870 22490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:47.769870 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:47.770066 22482 server_base.cc:1034] running on GCE node
I20260928 09:08:47.770413 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:47.770682 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:47.771828 22482 hybrid_clock.cc:648] HybridClock initialized: now 1790586527771807 us; error 43 us; skew 500 ppm
Sep 28 09:08:47 dist-test-slave-smns krb5kdc[22469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:47.779567 22482 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:47.780030 22482 webserver.cc:460] Webserver started at http://127.0.98.62:39833/ using document root <none> and password file <none>
I20260928 09:08:47.780267 22482 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:47.780318 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:47.780458 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:47.781626 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "7b2011de19114b99aa6c623c630459c5"
format_stamp: "Formatted at 2026-09-28 09:08:47 on dist-test-slave-smns"
server_key: "637b6e2a398900862f1dd803893794b6"
server_key_iv: "a2c2e540658ecd5a167a10abad8fbab9"
server_key_version: "encryptionkey@0"
I20260928 09:08:47.782002 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b2011de19114b99aa6c623c630459c5"
format_stamp: "Formatted at 2026-09-28 09:08:47 on dist-test-slave-smns"
server_key: "637b6e2a398900862f1dd803893794b6"
server_key_iv: "a2c2e540658ecd5a167a10abad8fbab9"
server_key_version: "encryptionkey@0"
I20260928 09:08:47.783341 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:08:47.784143 22497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:47.784332 22482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:08:47.784464 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "7b2011de19114b99aa6c623c630459c5"
format_stamp: "Formatted at 2026-09-28 09:08:47 on dist-test-slave-smns"
server_key: "637b6e2a398900862f1dd803893794b6"
server_key_iv: "a2c2e540658ecd5a167a10abad8fbab9"
server_key_version: "encryptionkey@0"
I20260928 09:08:47.784543 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:47.787973 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:47.789901 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:47.790045 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:47.793989 22482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37203
I20260928 09:08:47.794041 22549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37203 every 8 connection(s)
I20260928 09:08:47.794361 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:47.795389 22550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:47.797704 22550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5: Bootstrap starting.
I20260928 09:08:47.798280 22550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:47.798516 22550 log.cc:824] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:47.798812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22482
I20260928 09:08:47.798928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:47.799104 392 external_mini_cluster.cc:1428] Setting key 4951440013a32aac0537f229a31dbe9c
I20260928 09:08:47.799386 22550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5: No bootstrap required, opened a new log
I20260928 09:08:47.801247 22550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2011de19114b99aa6c623c630459c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260928 09:08:47.801437 22550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:08:47.801479 22550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2011de19114b99aa6c623c630459c5, State: Initialized, Role: FOLLOWER
I20260928 09:08:47.802016 22550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2011de19114b99aa6c623c630459c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260928 09:08:47.802152 22550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:08:47.802192 22550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:08:47.802251 22550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:47.802769 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:47.799626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49995 (local address 127.0.98.62:37203)
0928 09:08:47.799806 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:47.799815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:47.799858 (+ 43us) server_negotiation.cc:407] Connection header received
0928 09:08:47.801230 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:47.801238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:47.801272 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:47.801411 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:47.802188 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:47.802191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:47.802192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:47.802196 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:47.802264 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:47.802279 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:47.802356 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:47.802473 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 09:08:47.802604 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 09:08:47.803395 22550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2011de19114b99aa6c623c630459c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260928 09:08:47.803510 22550 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2011de19114b99aa6c623c630459c5; no voters:
I20260928 09:08:47.803681 22550 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:08:47.803740 22555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:47.803817 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:47.803917 22555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2011de19114b99aa6c623c630459c5, State: Running, Role: LEADER
I20260928 09:08:47.803943 22550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:47.804000 22555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2011de19114b99aa6c623c630459c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260928 09:08:47.804315 22557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2011de19114b99aa6c623c630459c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2011de19114b99aa6c623c630459c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } } }
I20260928 09:08:47.804394 22557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:47.804301 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2011de19114b99aa6c623c630459c5. Latest consensus state: current_term: 1 leader_uuid: "7b2011de19114b99aa6c623c630459c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2011de19114b99aa6c623c630459c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } } }
I20260928 09:08:47.804558 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5 [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:47.804672 22562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:47.805438 22562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:47.807042 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:47.805602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55685 (local address 127.0.98.62:37203)
0928 09:08:47.805640 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:47.805642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:47.805795 (+ 153us) server_negotiation.cc:407] Connection header received
0928 09:08:47.805888 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:47.805889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:47.805894 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:47.805939 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:47.806600 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:47.806602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:47.806602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:08:47.806606 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:47.806623 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:08:47.806628 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:47.806690 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:47.806965 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 09:08:47.806980 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:08:47.807528 22562 catalog_manager.cc:1347] Generated new cluster ID: 85168e31dd0d44c5a263035e7a93b0a5
I20260928 09:08:47.807578 22562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:47.836213 22562 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:47.836423 22562 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:47.840240 22562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2011de19114b99aa6c623c630459c5: Generated new TSK 0
I20260928 09:08:47.840459 22562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:08:47.846984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22482
2026-09-28T09:08:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1337 ms)
[----------] 18 tests from AuthTokenIssuingTest (24105 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:47 dist-test-slave-smns krb5kdc[22581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:47 dist-test-slave-smns krb5kdc[22581](info): set up 2 sockets
Sep 28 09:08:47 dist-test-slave-smns krb5kdc[22581](info): commencing operation
krb5kdc: starting...
W20260928 09:08:48.893884 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:48 dist-test-slave-smns krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:49.021765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46205,127.0.98.61:35947,127.0.98.60:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:49.105751 22594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:08:49.105973 22594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:49.105990 22594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:49.106012 22594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:49.107316 22594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:49.107364 22594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:49.107379 22594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:49.107393 22594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:49.107404 22594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:49.108958 22594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46205,127.0.98.61:35947,127.0.98.60:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:49.109242 22594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:49.109519 22594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:49.114665 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:49.114681 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:49.114800 22594 server_base.cc:1034] running on GCE node
W20260928 09:08:49.114671 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:49.115232 22594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:49.115533 22594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:49.116689 22594 hybrid_clock.cc:648] HybridClock initialized: now 1790586529116673 us; error 39 us; skew 500 ppm
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:49.119565 22594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:49.120069 22594 webserver.cc:460] Webserver started at http://127.0.98.62:33169/ using document root <none> and password file <none>
I20260928 09:08:49.120421 22594 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:49.120478 22594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:49.120633 22594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:49.121944 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "eb58aa43decd4a62a4303d5bc8871d0b"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "c5130cb34eba106d324f5ccace3c3ea3"
server_key_iv: "28d417a83a96fd9337283683cc90b0a6"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.122344 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb58aa43decd4a62a4303d5bc8871d0b"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "c5130cb34eba106d324f5ccace3c3ea3"
server_key_iv: "28d417a83a96fd9337283683cc90b0a6"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.123869 22594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:49.124848 22609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:49.125077 22594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:08:49.125170 22594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "eb58aa43decd4a62a4303d5bc8871d0b"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "c5130cb34eba106d324f5ccace3c3ea3"
server_key_iv: "28d417a83a96fd9337283683cc90b0a6"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.125233 22594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:49.137115 22594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:49.139428 22594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:49.139556 22594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:49.143661 22594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46205
I20260928 09:08:49.143759 22661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46205 every 8 connection(s)
I20260928 09:08:49.144126 22594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:49.145193 22662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:49.146926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22594
I20260928 09:08:49.147045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:49.147239 392 external_mini_cluster.cc:1428] Setting key ef39269964903a47186576e0e4161489
I20260928 09:08:49.147045 22662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 09:08:49.158175 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.147881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46205 (local address 127.0.0.1:48270)
0928 09:08:49.148109 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.148154 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.148849 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.149168 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.149171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.149426 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.149648 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.149652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.150731 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.150733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.151972 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.151975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.152554 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.152555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.152648 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.153205 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.153217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.154898 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.156735 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.156741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.156743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.156949 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.157166 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.157167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.157167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.157251 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.157672 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.157678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.157738 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.157850 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.158065 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":84,"thread_start_us":96,"threads_started":1}
I20260928 09:08:49.158179 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.147970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.98.62:46205)
0928 09:08:49.148132 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.148141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.148262 (+ 121us) server_negotiation.cc:407] Connection header received
0928 09:08:49.149018 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.149021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.149044 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.149154 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.149740 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.150626 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.152084 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.152425 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.152527 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.155008 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.155015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.155016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.155030 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.156659 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.157022 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.157024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.157024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.157102 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.157317 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.157318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.157318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.157590 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.157687 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.157886 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.157985 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 09:08:49.159058 22662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:49.159499 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.159315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40674)
0928 09:08:49.159406 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.159463 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 09:08:49.159672 22613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
I20260928 09:08:49.160142 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.160023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40688)
0928 09:08:49.160081 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.160111 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 09:08:49.160308 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35947: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 09:08:49.177908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35947
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46205,127.0.98.61:35947,127.0.98.60:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:08:49.209635 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } attempt: 1
I20260928 09:08:49.210196 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.210015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40690)
0928 09:08:49.210094 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.210154 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 09:08:49.210654 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.210555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40696)
0928 09:08:49.210601 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.210632 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:08:49.210758 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35947: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
W20260928 09:08:49.261119 22669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:08:49.261339 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:49.261358 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:49.261379 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:49.262493 22669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:49.262531 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:49.262545 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:49.262558 22669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:49.262570 22669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:49.263981 22669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46205,127.0.98.61:35947,127.0.98.60:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:49.264184 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:49.264364 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:49.269623 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:49.269632 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:49.269629 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:49.269667 22669 server_base.cc:1034] running on GCE node
I20260928 09:08:49.270215 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:49.270531 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:49.271673 22669 hybrid_clock.cc:648] HybridClock initialized: now 1790586529271654 us; error 42 us; skew 500 ppm
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:49.274300 22669 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 09:08:49.274771 22669 webserver.cc:460] Webserver started at http://127.0.98.61:39213/ using document root <none> and password file <none>
I20260928 09:08:49.274955 22669 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:49.274994 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:49.275082 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:49.276115 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data/instance:
uuid: "52dcd5dd31194df891ac23221fc64580"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "1525f00eedd390f625bdea9925844113"
server_key_iv: "92fa3a8bfa4f54a7cee29eef0236a45a"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.276423 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal/instance:
uuid: "52dcd5dd31194df891ac23221fc64580"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "1525f00eedd390f625bdea9925844113"
server_key_iv: "92fa3a8bfa4f54a7cee29eef0236a45a"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.277829 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:49.278702 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:49.278929 22669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:49.278905 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } attempt: 2
I20260928 09:08:49.279035 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal
uuid: "52dcd5dd31194df891ac23221fc64580"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "1525f00eedd390f625bdea9925844113"
server_key_iv: "92fa3a8bfa4f54a7cee29eef0236a45a"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.279120 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:49.279394 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.279200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40710)
0928 09:08:49.279322 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.279357 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 09:08:49.279763 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.279648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40722)
0928 09:08:49.279707 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.279738 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 09:08:49.279876 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35947: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35947: connect: Connection refused (error 111)
I20260928 09:08:49.296656 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:49.298939 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:49.299095 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:49.303356 22669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35947
I20260928 09:08:49.303422 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35947 every 8 connection(s)
I20260928 09:08:49.303761 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:08:49.304762 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:49.306461 22737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:49.313031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22669
I20260928 09:08:49.313122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-1/wal/instance
I20260928 09:08:49.313313 392 external_mini_cluster.cc:1428] Setting key 3f0fda24c7f9badc0f97c0b30fae6b39
I20260928 09:08:49.315976 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.307187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.62:46205)
0928 09:08:49.307249 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.307252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.307486 (+ 234us) server_negotiation.cc:407] Connection header received
0928 09:08:49.308022 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.308024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.308036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.308101 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.308720 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.309201 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.310320 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.310533 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.310642 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.312967 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.312968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.312969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.312976 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.314534 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.314999 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.315000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.315000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.315061 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.315322 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.315323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.315323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.315439 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.315510 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.315803 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.315883 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:08:49.316174 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.307165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46205 (local address 127.0.0.1:48280)
0928 09:08:49.307348 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.307388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.307911 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.308181 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.308184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.308433 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.308619 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.308623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.309312 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.309313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.310229 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.310231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.310661 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.310663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.310824 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.311429 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.311437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.312856 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.314645 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.314652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.314654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.314916 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.315130 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.315131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.315131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.315252 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.315518 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.315526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.315664 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.315770 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.315929 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260928 09:08:49.316757 22737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:49.328635 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.317048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40736)
0928 09:08:49.317102 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.317113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.317155 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.317424 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.317425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.317624 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.317676 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.317677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.318561 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.318563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.319096 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.319097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.319612 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.319613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.319677 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.320195 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.320200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.324771 (+ 4571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.327302 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.327303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.327303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.327499 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.327757 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.327757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.327757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.327810 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.328328 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.328329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.328393 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.328463 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.328560 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:08:49.328783 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.317128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.0.98.61:35947)
0928 09:08:49.317287 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.317297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.317316 (+ 19us) server_negotiation.cc:407] Connection header received
0928 09:08:49.317332 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.317334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.317344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.317424 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.317787 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.318463 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.319195 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.319521 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.319613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.324903 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.324918 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.324919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.324928 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.327185 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.327592 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.327595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.327596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.327666 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.327876 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.327877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.327878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.328118 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.328225 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.328537 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.328631 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260928 09:08:49.329587 22737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:49.330019 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.329870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40220)
0928 09:08:49.329944 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.329996 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 09:08:49.330209 22688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
I20260928 09:08:49.330679 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.330582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40234)
0928 09:08:49.330636 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.330660 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 09:08:49.330816 22737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 09:08:49.341645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41619
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46205,127.0.98.61:35947,127.0.98.60:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:08:49.371068 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } attempt: 3
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:49.378500 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.371548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.98.61:35947)
0928 09:08:49.371614 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.371616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.371627 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:08:49.371678 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.371679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.371695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.371759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.372162 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.372652 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.373416 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.373679 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.373774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.375686 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.375687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.375688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.375696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.377101 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.377481 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.377482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.377482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.377587 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.377813 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.377814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.377814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.378035 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.378146 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.378306 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.378398 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:08:49.378500 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.371477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40738)
0928 09:08:49.371541 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.371551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.371612 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.371778 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.371779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.371970 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.372046 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.372048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.372748 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.372749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.373281 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.373283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.373774 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.373775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.373861 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.374366 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.374372 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.375590 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.377194 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.377195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.377196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.377386 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.377664 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.377665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.377666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.377734 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.378138 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.378139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.378190 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.378281 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.378391 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 09:08:49.378865 22662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:49.379235 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.379125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40250)
0928 09:08:49.379178 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.379206 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:08:49.379597 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.379513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40256)
0928 09:08:49.379562 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.379584 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 09:08:49.379700 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
I20260928 09:08:49.380049 22737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } attempt: 1
I20260928 09:08:49.380457 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.380342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40272)
0928 09:08:49.380397 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.380427 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:08:49.380807 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.380714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40288)
0928 09:08:49.380761 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.380784 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:08:49.380901 22737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
I20260928 09:08:49.410866 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } attempt: 1
I20260928 09:08:49.411401 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.411250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40298)
0928 09:08:49.411324 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.411368 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 09:08:49.411777 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.411656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40302)
0928 09:08:49.411724 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.411757 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 09:08:49.411873 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
W20260928 09:08:49.428679 22744 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:08:49.428908 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:49.428925 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:49.428982 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:49.430208 22744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:49.430260 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:49.430280 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:49.430294 22744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:49.430305 22744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:49.431816 22744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46205,127.0.98.61:35947,127.0.98.60:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:49.432089 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:49.432355 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:49.437633 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:49.437633 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:49.437666 22744 server_base.cc:1034] running on GCE node
W20260928 09:08:49.437664 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:49.438113 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:49.438424 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:49.439604 22744 hybrid_clock.cc:648] HybridClock initialized: now 1790586529439584 us; error 35 us; skew 500 ppm
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:49.442401 22744 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 09:08:49.442932 22744 webserver.cc:460] Webserver started at http://127.0.98.60:39689/ using document root <none> and password file <none>
I20260928 09:08:49.443181 22744 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:49.443252 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:49.443424 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:49.444535 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data/instance:
uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "da0756691c06dc10e9a6e2d94224ba86"
server_key_iv: "d5eee9fa6c166f2e262f38496cf0d496"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.444918 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal/instance:
uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "da0756691c06dc10e9a6e2d94224ba86"
server_key_iv: "d5eee9fa6c166f2e262f38496cf0d496"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.446527 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:08:49.447342 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:49.447562 22744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:08:49.447680 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal
uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e"
format_stamp: "Formatted at 2026-09-28 09:08:49 on dist-test-slave-smns"
server_key: "da0756691c06dc10e9a6e2d94224ba86"
server_key_iv: "d5eee9fa6c166f2e262f38496cf0d496"
server_key_version: "encryptionkey@0"
I20260928 09:08:49.447769 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:49.449036 22737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } attempt: 2
I20260928 09:08:49.449426 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.449302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40308)
0928 09:08:49.449358 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.449392 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 09:08:49.449748 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.449659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40324)
0928 09:08:49.449703 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.449727 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 09:08:49.449836 22737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41619: connect: Connection refused (error 111)
I20260928 09:08:49.465278 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:49.467753 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:49.467931 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:49.472318 22744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41619
I20260928 09:08:49.472371 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41619 every 8 connection(s)
I20260928 09:08:49.472749 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:08:49.473912 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:49.475466 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:49.476962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22744
I20260928 09:08:49.477098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790586261566264-392-0/minicluster-data/master-2/wal/instance
I20260928 09:08:49.477274 392 external_mini_cluster.cc:1428] Setting key f02d7c43362cf63ac38cc8f3680e90ac
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:08:49.486877 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.476309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.98.62:46205)
0928 09:08:49.476385 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.476387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.476497 (+ 110us) server_negotiation.cc:407] Connection header received
0928 09:08:49.477045 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.477047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.477059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.477168 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.478261 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.478748 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.480199 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.480514 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.480604 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.483182 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.483184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.483185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.483194 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.484850 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.485570 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.485571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.485572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.485640 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.485999 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.486000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.486000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.486133 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.486281 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.486676 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.486759 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:08:49.486951 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.476187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46205 (local address 127.0.0.1:48292)
0928 09:08:49.476377 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.476419 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.476929 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.477253 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.477257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.477764 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.478065 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.478070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.478848 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.478849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.480066 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.480069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.480635 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.480637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.480767 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.481313 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.481323 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.483082 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.484994 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.485002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.485005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.485475 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.485726 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.485727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.485727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.485896 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.486293 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.486299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.486488 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.486603 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.486744 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260928 09:08:49.486989 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } attempt: 2
I20260928 09:08:49.487952 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:49.488289 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.477727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.98.60:41619)
0928 09:08:49.477922 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.477927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.477977 (+ 50us) server_negotiation.cc:407] Connection header received
0928 09:08:49.478037 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.478040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.478061 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.478137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.478919 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.480108 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.481066 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.481623 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.481700 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.484523 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.484531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.484531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.484538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.486511 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.486971 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.486974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.486974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.487034 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.487299 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.487300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.487300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.487455 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.487618 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.488078 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.488166 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":164,"thread_start_us":131,"threads_started":1}
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:49.496795 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.487671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40344)
0928 09:08:49.487723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.487733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.487772 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.487996 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.487997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.488161 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.488216 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.488218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.489324 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.489325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.490165 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.490167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.490643 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.490644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.490714 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.491188 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.491194 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.492867 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.495647 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.495649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.495649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.495889 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.496087 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.496088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.496144 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.496431 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.496432 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.496478 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.496552 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.496683 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 09:08:49.496944 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.488277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.0.98.61:35947)
0928 09:08:49.488350 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.488353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.488587 (+ 234us) server_negotiation.cc:407] Connection header received
0928 09:08:49.488650 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.488651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.488662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.488718 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.489024 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.489785 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.490750 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.491072 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.491144 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.493646 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.493648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.493648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.493656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.495657 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496054 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.496056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.496056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.496112 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496312 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.496313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.496314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.496444 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.496505 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.496685 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.496764 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:08:49.496995 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.489133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.0.98.62:46205)
0928 09:08:49.489185 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.489187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.489193 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:08:49.489210 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.489211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.489219 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.489274 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.489856 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.490316 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.491220 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.491516 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.491590 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.494181 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.494182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.494182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.494189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.495634 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496010 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.496012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.496012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.496068 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496264 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.496265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.496265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.496399 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.496594 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.496814 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.496895 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:08:49.497200 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.488216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35947 (local address 127.0.0.1:40742)
0928 09:08:49.488535 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.488544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.488597 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.488721 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.488722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.488870 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.488946 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.488948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.489882 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.489882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.490671 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.490672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.491850 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.491852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.491938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.492255 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.492260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.493528 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.495741 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.495742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.495742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.495946 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.496168 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.496169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.496224 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.496498 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.496499 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.496539 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.497047 (+ 508us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.497144 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260928 09:08:49.497573 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:49.497627 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.487715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.98.60:41619)
0928 09:08:49.487865 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.487867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.487874 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:08:49.487894 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.487895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.487905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.488314 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.488353 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.489057 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.490253 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.490564 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.490893 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.492982 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.492983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.492983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.492989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.495542 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.495975 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.495976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.495977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.496037 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.496214 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.496214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.496215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.496346 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.496440 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.496591 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.496696 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260928 09:08:49.499738 22662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b: Bootstrap starting.
I20260928 09:08:49.500469 22662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:49.500808 22662 log.cc:824] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:49.501842 22662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b: No bootstrap required, opened a new log
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:49.503782 22662 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.503942 22662 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb58aa43decd4a62a4303d5bc8871d0b, State: Initialized, Role: FOLLOWER
I20260928 09:08:49.504036 22662 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.504416 22662 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:49.504384 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.504520 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:08:49.506253 22830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:49.506352 22830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:49.506435 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.497942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40348)
0928 09:08:49.497993 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.498069 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.498117 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.498351 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.498352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.498489 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.498517 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.498518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.499263 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.499263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.499963 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.499965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.500978 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.500979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.501042 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.501928 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.501933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.503409 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.505189 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.505190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.505191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.505411 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.505670 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.505672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.505672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.505738 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.506091 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.506092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.506135 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.506200 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.506302 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:08:49.506831 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.498007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.0.98.60:41619)
0928 09:08:49.498041 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.498043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.498104 (+ 61us) server_negotiation.cc:407] Connection header received
0928 09:08:49.498276 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.498277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.498286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.498337 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.498591 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.499180 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.500047 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.500460 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.500581 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.503508 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.503510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.503510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.503517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.505100 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.505481 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.505482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.505482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.505587 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.505806 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.505807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.505807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.506000 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.506066 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.506621 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.506717 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:08:49.508612 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: Bootstrap starting.
I20260928 09:08:49.508880 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.498221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.61:35947)
0928 09:08:49.498276 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.498278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.498286 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:49.498306 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.498307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.498317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.498368 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.498991 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.499533 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.500547 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.500845 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.500922 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.505709 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.505711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.505712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.505719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.507504 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.507913 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.507914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.507914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.507969 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.508183 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.508184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.508184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.508355 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.508480 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.508708 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.508787 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:08:49.509411 22812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:49.507105 22594 master_runner.cc:186] Error getting master registration for 127.0.98.61:35947: OK, instance_id { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" instance_seqno: 1790586529302100 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:08:49.509663 22594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:08:49.509769 22812 log.cc:824] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:49.510607 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: No bootstrap required, opened a new log
I20260928 09:08:49.510629 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:49.511670 22636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eb58aa43decd4a62a4303d5bc8871d0b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48300
I20260928 09:08:49.511821 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 09:08:49.511883 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:49.512844 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.512779 22812 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.512923 22812 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb3a2f1c4b4f10bd8b62917dc98a6e, State: Initialized, Role: FOLLOWER
I20260928 09:08:49.512997 22812 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.513067 22636 leader_election.cc:288] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [CANDIDATE]: Term 1 election: Requested vote from peers 52dcd5dd31194df891ac23221fc64580 (127.0.98.61:35947), ecdb3a2f1c4b4f10bd8b62917dc98a6e (127.0.98.60:41619)
I20260928 09:08:49.513294 22812 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:49.513382 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.513485 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:49.513859 22786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e"
I20260928 09:08:49.513988 22786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:49.515285 22786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb58aa43decd4a62a4303d5bc8871d0b in term 1.
I20260928 09:08:49.515657 22613 leader_election.cc:302] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb58aa43decd4a62a4303d5bc8871d0b, ecdb3a2f1c4b4f10bd8b62917dc98a6e; no voters:
W20260928 09:08:49.516144 22844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:49.516201 22844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:49.516731 22819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:49.516935 22819 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [term 1 LEADER]: Becoming Leader. State: Replica: eb58aa43decd4a62a4303d5bc8871d0b, State: Running, Role: LEADER
I20260928 09:08:49.517027 22819 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.517616 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb58aa43decd4a62a4303d5bc8871d0b. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.517724 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:49.517939 22853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:49.518132 22853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:49.519644 22786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [term 1 FOLLOWER]: Refusing update from remote peer eb58aa43decd4a62a4303d5bc8871d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:49.517582 22744 master_runner.cc:186] Error getting master registration for 127.0.98.61:35947: OK, instance_id { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" instance_seqno: 1790586529302100 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:08:49.519755 22744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:08:49.520179 22819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:49.521862 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb58aa43decd4a62a4303d5bc8871d0b. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.521950 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:49.530416 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.517445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.98.61:35947)
0928 09:08:49.517487 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.517489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.520354 (+ 2865us) server_negotiation.cc:407] Connection header received
0928 09:08:49.520393 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.520394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.520420 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.520489 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.521240 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.522027 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.523116 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.523452 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.523510 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.525317 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.525319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.525319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.525328 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.527005 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.527488 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.527490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.527491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.527566 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.528043 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.528044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.528045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.529331 (+ 1286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.529942 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.530187 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.530308 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:08:49.531504 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.516860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.98.62:46205)
0928 09:08:49.516906 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.516909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.520106 (+ 3197us) server_negotiation.cc:407] Connection header received
0928 09:08:49.520217 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.520218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.520228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.520259 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.521607 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.522320 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.524058 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.524371 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.524410 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.527306 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.527308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.527308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.527317 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.529147 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.530007 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.530008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.530009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.530083 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.530317 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.530317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.530318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.530576 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.530686 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.531294 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.531392 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:49.533429 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.520133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.0.98.60:41619)
0928 09:08:49.520175 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.520177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.522165 (+ 1988us) server_negotiation.cc:407] Connection header received
0928 09:08:49.522293 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.522294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.522304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.522334 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.523670 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.524411 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.526112 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.526436 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.526486 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.529418 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.529420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.529421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.529429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.531510 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.532392 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.532393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.532394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.532475 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.532754 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.532755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.532755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.532888 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.532990 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.533197 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.533305 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:49.538708 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.538867 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:49.538940 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.539013 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:49.539038 22857 mvcc.cc:205] Tried to move back new op lower bound from 7334242424910102528 to 7334242424902828032. Current Snapshot: MvccSnapshot[applied={T|T < 7334242424910102528}]
I20260928 09:08:49.539968 22853 catalog_manager.cc:1347] Generated new cluster ID: 6d06ad7f71ca4feabd5b1bfcd44a1239
I20260928 09:08:49.540024 22853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:49.545756 22737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } attempt: 3
Sep 28 09:08:49 dist-test-slave-smns krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:08:49.555544 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.546259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41619 (local address 127.0.0.1:40364)
0928 09:08:49.546370 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:49.546382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:49.546426 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:49.546685 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:49.546686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:49.546911 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:49.546972 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.546974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.547935 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.547936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.548767 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:49.548770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.549252 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.549253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:49.549311 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.550244 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:49.550249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:49.552012 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:49.554017 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.554018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.554019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.554254 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.554572 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:49.554573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:49.554573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:49.554659 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:49.555068 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:49.555069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:49.555131 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 09:08:49.555190 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 09:08:49.555434 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 09:08:49.555991 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:49.546389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.98.60:41619)
0928 09:08:49.546450 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.546452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.546464 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:08:49.546499 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.546500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.546511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.546582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.547052 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.547841 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.548856 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:49.549162 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:49.549240 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:49.552094 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:49.552096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:49.552096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:49.552105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:49.553930 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.554320 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.554321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.554322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.554436 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:49.554738 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:49.554738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:49.554739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:49.554942 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:49.555656 (+ 714us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:49.555730 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 09:08:49.555850 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:08:49.558005 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: Bootstrap starting.
I20260928 09:08:49.558811 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:49.559137 22737 log.cc:824] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:49.565474 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: No bootstrap required, opened a new log
I20260928 09:08:49.567816 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.568029 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dcd5dd31194df891ac23221fc64580, State: Initialized, Role: FOLLOWER
I20260928 09:08:49.568130 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } }
I20260928 09:08:49.568521 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:49.568625 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:49.569664 22710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [term 1 FOLLOWER]: Refusing update from remote peer eb58aa43decd4a62a4303d5bc8871d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:49.569931 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "52dcd5dd31194df891ac23221fc64580"
I20260928 09:08:49.570446 22860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:49.570730 22711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb58aa43decd4a62a4303d5bc8871d0b in term 1.
I20260928 09:08:49.574482 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.574594 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b [sys.catalog]: This master's current role is: LEADER
W20260928 09:08:49.575168 22878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:49.575295 22878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:49.575330 22853 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:49.576086 22853 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:49.579332 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.579452 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:49.579510 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb58aa43decd4a62a4303d5bc8871d0b. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.579551 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:49.579589 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb58aa43decd4a62a4303d5bc8871d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46205 } } peers { permanent_uuid: "52dcd5dd31194df891ac23221fc64580" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35947 } } peers { permanent_uuid: "ecdb3a2f1c4b4f10bd8b62917dc98a6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41619 } } }
I20260928 09:08:49.579631 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:49.580211 22875 mvcc.cc:205] Tried to move back new op lower bound from 7334242424910102528 to 7334242424902828032. Current Snapshot: MvccSnapshot[applied={T|T < 7334242424910102528}]
I20260928 09:08:49.580479 22853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb58aa43decd4a62a4303d5bc8871d0b: Generated new TSK 0
I20260928 09:08:49.580677 22853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:08:49.610548 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.609151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.98.61:35947)
0928 09:08:49.609207 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.609210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.609826 (+ 616us) server_negotiation.cc:407] Connection header received
0928 09:08:49.609936 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.609937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.609948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.609979 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.610470 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40758: BlockingRecv error: recv got EOF from 127.0.0.1:40758 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 09:08:49.611517 22665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.608911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.0.98.62:46205)
0928 09:08:49.608958 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.608960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.610964 (+ 2004us) server_negotiation.cc:407] Connection header received
0928 09:08:49.611085 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.611086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.611097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.611136 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.611454 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48318: BlockingRecv error: recv got EOF from 127.0.0.1:48318 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:08:49.612356 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.609415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.98.60:41619)
0928 09:08:49.609455 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.609457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.611863 (+ 2406us) server_negotiation.cc:407] Connection header received
0928 09:08:49.611957 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.611958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.611969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.611997 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.612295 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40376: BlockingRecv error: recv got EOF from 127.0.0.1:40376 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 09:08:49.614908 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.614180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.0.98.61:35947)
0928 09:08:49.614232 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.614234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.614358 (+ 124us) server_negotiation.cc:407] Connection header received
0928 09:08:49.614464 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.614465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.614485 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.614516 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.614862 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40762: BlockingRecv error: recv got EOF from 127.0.0.1:40762 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 09:08:49.618093 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.617284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.98.61:35947)
0928 09:08:49.617319 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.617320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.617609 (+ 289us) server_negotiation.cc:407] Connection header received
0928 09:08:49.617708 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.617709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.617720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.617750 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.618047 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40770: BlockingRecv error: recv got EOF from 127.0.0.1:40770 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 09:08:49.622242 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.621421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.98.61:35947)
0928 09:08:49.621459 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.621461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.621710 (+ 249us) server_negotiation.cc:407] Connection header received
0928 09:08:49.621810 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.621811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.621821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.621849 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.622184 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40778: BlockingRecv error: recv got EOF from 127.0.0.1:40778 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 09:08:49.628435 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.627528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.0.98.61:35947)
0928 09:08:49.627565 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.627567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.627855 (+ 288us) server_negotiation.cc:407] Connection header received
0928 09:08:49.627963 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.627964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.627979 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.628011 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.628372 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40786: BlockingRecv error: recv got EOF from 127.0.0.1:40786 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 09:08:49.638964 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.637989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.0.98.61:35947)
0928 09:08:49.638036 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.638039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.638312 (+ 273us) server_negotiation.cc:407] Connection header received
0928 09:08:49.638423 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.638424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.638436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.638467 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.638888 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40790: BlockingRecv error: recv got EOF from 127.0.0.1:40790 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:08:49.657495 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.656505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.98.61:35947)
0928 09:08:49.656550 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.656552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.656856 (+ 304us) server_negotiation.cc:407] Connection header received
0928 09:08:49.656967 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.656968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.656980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.657009 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.657428 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40792: BlockingRecv error: recv got EOF from 127.0.0.1:40792 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 09:08:49.691901 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.690959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.98.61:35947)
0928 09:08:49.691004 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.691006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.691284 (+ 278us) server_negotiation.cc:407] Connection header received
0928 09:08:49.691390 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.691391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.691402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.691432 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.691836 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40806: BlockingRecv error: recv got EOF from 127.0.0.1:40806 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 09:08:49.758157 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.757470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.98.61:35947)
0928 09:08:49.757522 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.757524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.757637 (+ 113us) server_negotiation.cc:407] Connection header received
0928 09:08:49.757698 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.757699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.757710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.757792 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.758092 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40810: BlockingRecv error: recv got EOF from 127.0.0.1:40810 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 09:08:49.888163 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:49.887336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.98.61:35947)
0928 09:08:49.887431 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:49.887434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:49.887534 (+ 100us) server_negotiation.cc:407] Connection header received
0928 09:08:49.887650 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:49.887651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:49.887663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:49.887738 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:49.888095 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40822: BlockingRecv error: recv got EOF from 127.0.0.1:40822 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 09:08:50.146170 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:50.145353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.0.98.61:35947)
0928 09:08:50.145425 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:50.145427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:50.145527 (+ 100us) server_negotiation.cc:407] Connection header received
0928 09:08:50.145655 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:50.145657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:50.145669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:50.145749 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:50.146129 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40832: BlockingRecv error: recv got EOF from 127.0.0.1:40832 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:08:50.516634 22844 catalog_manager.cc:1259] Loaded cluster ID: 6d06ad7f71ca4feabd5b1bfcd44a1239
I20260928 09:08:50.516703 22844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: loading cluster ID for follower catalog manager: success
I20260928 09:08:50.517532 22844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: acquiring CA information for follower catalog manager: success
I20260928 09:08:50.517930 22844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecdb3a2f1c4b4f10bd8b62917dc98a6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:08:50.575845 22878 catalog_manager.cc:1259] Loaded cluster ID: 6d06ad7f71ca4feabd5b1bfcd44a1239
I20260928 09:08:50.575918 22878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: loading cluster ID for follower catalog manager: success
I20260928 09:08:50.576794 22878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: acquiring CA information for follower catalog manager: success
I20260928 09:08:50.577137 22878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52dcd5dd31194df891ac23221fc64580: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:08:50.662371 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:50.659464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.0.98.61:35947)
0928 09:08:50.659566 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:50.659568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:50.659597 (+ 29us) server_negotiation.cc:407] Connection header received
0928 09:08:50.659704 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:50.659705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:50.659759 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:50.659830 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:50.659989 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.660549 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.661462 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.661694 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.661775 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:50.661966 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:50.662089 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:50.662183 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:50.662237 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 09:08:50.662263 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260928 09:08:50.667095 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:50.664468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.0.98.60:41619)
0928 09:08:50.664568 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:50.664570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:50.664578 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:50.664636 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:50.664636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:50.664672 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:50.664738 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:50.664844 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.665330 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.666165 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.666474 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.666544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:50.666658 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:50.666794 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:50.666854 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:50.666960 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 09:08:50.666981 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
I20260928 09:08:50.672379 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:50.669366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.98.60:41619)
0928 09:08:50.669457 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:50.669460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:50.669882 (+ 422us) server_negotiation.cc:407] Connection header received
0928 09:08:50.669916 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:50.669917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:50.669927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:50.669979 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:50.670117 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.670827 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.671593 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.671885 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.671947 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:50.672130 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:50.672167 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:50.672216 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:50.672270 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 09:08:50.672291 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 09:08:50.672482 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:50.669089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.0.98.61:35947)
0928 09:08:50.669146 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:50.669148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:50.669363 (+ 215us) server_negotiation.cc:407] Connection header received
0928 09:08:50.669509 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:50.669510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:50.669519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:50.669615 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:50.669812 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.670596 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.671655 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.671943 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.672000 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:50.672175 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:50.672207 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:50.672255 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:50.672378 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 09:08:50.672407 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 09:08:50.672749 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:50.669228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.0.98.62:46205)
0928 09:08:50.669372 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:50.669374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:50.669382 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:50.669429 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:50.669430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:50.669485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:50.669644 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:50.669746 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.670507 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.671642 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:50.671871 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:50.671963 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:50.672166 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:50.672284 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:50.672594 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:50.672612 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 09:08:50.672637 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260928 09:08:50.674499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22594
I20260928 09:08:50.680167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22669
I20260928 09:08:50.685432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22744
2026-09-28T09:08:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2839 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:08:50 dist-test-slave-smns krb5kdc[22964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:08:50 dist-test-slave-smns krb5kdc[22964](info): set up 2 sockets
Sep 28 09:08:50 dist-test-slave-smns krb5kdc[22964](info): commencing operation
krb5kdc: starting...
W20260928 09:08:51.746141 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/test-user.keytab.
Sep 28 09:08:51 dist-test-slave-smns krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:08:51.869100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40881,127.0.98.61:33441,127.0.98.60:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:08:51.950269 22977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:08:51.950551 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:51.950580 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:51.950654 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:51.951874 22977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:51.951952 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:51.951977 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:51.951997 22977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:51.952028 22977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:51.953572 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40881,127.0.98.61:33441,127.0.98.60:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:51.953866 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:51.954107 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:51.958978 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:51.958971 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:51.958974 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:51.959208 22977 server_base.cc:1034] running on GCE node
I20260928 09:08:51.959549 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:51.959841 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:51.961010 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790586531960987 us; error 59 us; skew 500 ppm
Sep 28 09:08:51 dist-test-slave-smns krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:51.963729 22977 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:08:51.964180 22977 webserver.cc:460] Webserver started at http://127.0.98.62:33543/ using document root <none> and password file <none>
I20260928 09:08:51.964509 22977 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:51.964558 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:51.964761 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:51.965931 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data/instance:
uuid: "f302ee6a833a457cb754f2cdc62ad5db"
format_stamp: "Formatted at 2026-09-28 09:08:51 on dist-test-slave-smns"
server_key: "4973b36a07ddf02847f2db83cb84958b"
server_key_iv: "947d0842bb3691fba6ea9244d5db562b"
server_key_version: "encryptionkey@0"
I20260928 09:08:51.966300 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal/instance:
uuid: "f302ee6a833a457cb754f2cdc62ad5db"
format_stamp: "Formatted at 2026-09-28 09:08:51 on dist-test-slave-smns"
server_key: "4973b36a07ddf02847f2db83cb84958b"
server_key_iv: "947d0842bb3691fba6ea9244d5db562b"
server_key_version: "encryptionkey@0"
I20260928 09:08:51.967689 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:51.968426 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:51.968617 22977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:51.968710 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal
uuid: "f302ee6a833a457cb754f2cdc62ad5db"
format_stamp: "Formatted at 2026-09-28 09:08:51 on dist-test-slave-smns"
server_key: "4973b36a07ddf02847f2db83cb84958b"
server_key_iv: "947d0842bb3691fba6ea9244d5db562b"
server_key_version: "encryptionkey@0"
I20260928 09:08:51.968786 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:51.988520 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:51.991122 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:51.991384 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:51.995415 22977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40881
I20260928 09:08:51.995483 23044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40881 every 8 connection(s)
I20260928 09:08:51.995818 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:08:51.996799 23045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:51.998304 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:52.003971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 22977
I20260928 09:08:52.004091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-0/wal/instance
I20260928 09:08:52.004396 392 external_mini_cluster.cc:1428] Setting key 635999402df7da026dd8f1a9e1aebfa1
Sep 28 09:08:51 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:52.009063 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:51.998971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40881 (local address 127.0.0.1:42670)
0928 09:08:51.999128 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:51.999170 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:51.999614 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:51.999940 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:51.999942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.000178 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.000333 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.000336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.001449 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.001450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.002226 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.002229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.002778 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.002779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.002838 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.003524 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.003533 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.005177 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.007312 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.007319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.007321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.007605 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.007873 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.007874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.007875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.007993 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.008405 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.008413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.008497 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.008610 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.008844 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260928 09:08:52.009071 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:51.999020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.0.98.62:40881)
0928 09:08:51.999207 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:51.999211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:51.999250 (+ 39us) server_negotiation.cc:407] Connection header received
0928 09:08:51.999789 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:51.999791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:51.999808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:51.999897 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.000456 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.001356 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.002325 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.002678 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.002773 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.005269 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.005277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.005278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.005285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.007225 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.007714 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.007717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.007717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.007800 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.008065 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.008066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.008066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.008330 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.008396 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.008634 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.008772 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260928 09:08:52.009994 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:52.010409 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.010264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33684)
0928 09:08:52.010320 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.010376 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 09:08:52.010572 22996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
I20260928 09:08:52.011073 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.010966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33700)
0928 09:08:52.011016 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.011050 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:08:52.011193 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 09:08:52.032470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33441
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40881,127.0.98.61:33441,127.0.98.60:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:08:52.060499 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } attempt: 1
I20260928 09:08:52.061025 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.060883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33702)
0928 09:08:52.060950 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.060996 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 09:08:52.061419 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.061295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33704)
0928 09:08:52.061374 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.061398 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 09:08:52.061518 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
W20260928 09:08:52.114673 23052 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:08:52.114925 23052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:52.114987 23052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:52.115042 23052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:52.116182 23052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:52.116357 23052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:52.116441 23052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:52.116475 23052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:52.116505 23052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:52.118351 23052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40881,127.0.98.61:33441,127.0.98.60:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:52.118635 23052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:52.118870 23052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:52.123974 23060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:52.123967 23058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:52.124042 23052 server_base.cc:1034] running on GCE node
W20260928 09:08:52.123967 23057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:52.124612 23052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:52.124959 23052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:52.126120 23052 hybrid_clock.cc:648] HybridClock initialized: now 1790586532126090 us; error 43 us; skew 500 ppm
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:52.128945 23052 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 09:08:52.129395 23052 webserver.cc:460] Webserver started at http://127.0.98.61:46577/ using document root <none> and password file <none>
I20260928 09:08:52.129706 23052 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:52.129700 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } attempt: 2
I20260928 09:08:52.129776 23052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:52.129911 23052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:52.130089 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.129955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33716)
0928 09:08:52.130017 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.130049 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 09:08:52.130429 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.130326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33728)
0928 09:08:52.130379 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.130404 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 09:08:52.130559 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33441: connect: Connection refused (error 111)
I20260928 09:08:52.131186 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data/instance:
uuid: "532524efa57f42d5ad19bd03e77e28cf"
format_stamp: "Formatted at 2026-09-28 09:08:52 on dist-test-slave-smns"
server_key: "a56a9ea47ecce3e654ddac281480f1bd"
server_key_iv: "7f4b8082f400d6b9f8e36014a44987d4"
server_key_version: "encryptionkey@0"
I20260928 09:08:52.131575 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal/instance:
uuid: "532524efa57f42d5ad19bd03e77e28cf"
format_stamp: "Formatted at 2026-09-28 09:08:52 on dist-test-slave-smns"
server_key: "a56a9ea47ecce3e654ddac281480f1bd"
server_key_iv: "7f4b8082f400d6b9f8e36014a44987d4"
server_key_version: "encryptionkey@0"
I20260928 09:08:52.132874 23052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:08:52.133677 23067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:52.133848 23052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:52.133944 23052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal
uuid: "532524efa57f42d5ad19bd03e77e28cf"
format_stamp: "Formatted at 2026-09-28 09:08:52 on dist-test-slave-smns"
server_key: "a56a9ea47ecce3e654ddac281480f1bd"
server_key_iv: "7f4b8082f400d6b9f8e36014a44987d4"
server_key_version: "encryptionkey@0"
I20260928 09:08:52.134011 23052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:52.144860 23052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:52.146754 23052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:52.146914 23052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:52.150838 23052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33441
I20260928 09:08:52.150871 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33441 every 8 connection(s)
I20260928 09:08:52.151233 23052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:08:52.152215 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:52.153729 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:52.157346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 23052
I20260928 09:08:52.157447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-1/wal/instance
I20260928 09:08:52.157688 392 external_mini_cluster.cc:1428] Setting key 8f40b48e54e6c9cc7ef786023eaadb97
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:52.162863 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.154416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.0.98.62:40881)
0928 09:08:52.154485 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.154487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.154680 (+ 193us) server_negotiation.cc:407] Connection header received
0928 09:08:52.155104 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.155105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.155115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.155172 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.155733 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.156217 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.157116 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.157507 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.157669 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.159918 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.159920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.159920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.159926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.161309 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.161836 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.161836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.161837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.161905 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.162199 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.162199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.162200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.162302 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.162351 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.162707 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.162777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:08:52.163139 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.154403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40881 (local address 127.0.0.1:42684)
0928 09:08:52.154554 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.154594 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.155027 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.155220 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.155223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.155480 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.155660 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.155664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.156294 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.156295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.157047 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.157049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.157687 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.157688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.157832 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.158365 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.158374 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.159811 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.161393 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.161400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.161402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.161761 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.161976 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.161977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.161978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.162131 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.162368 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.162376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.162560 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.162683 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.162882 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 09:08:52.163825 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:52.172259 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.164111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33734)
0928 09:08:52.164170 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.164184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.164232 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.164529 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.164530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.164699 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.164767 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.164769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.165653 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.165654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.166224 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.166226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.166700 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.166701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.166764 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.167136 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.167140 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.169162 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.171097 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.171098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.171098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.171334 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.171567 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.171568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.171568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.171641 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.171938 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.171939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.171977 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.172041 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.172140 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 09:08:52.172322 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.164164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.98.61:33441)
0928 09:08:52.164330 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.164335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.164354 (+ 19us) server_negotiation.cc:407] Connection header received
0928 09:08:52.164383 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.164385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.164398 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.164510 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.164846 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.165584 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.166295 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.166620 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.166705 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.169244 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.169253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.169253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.169261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.171013 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.171449 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.171451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.171452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.171500 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.171703 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.171704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.171704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.171857 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.171960 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.172069 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.172192 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260928 09:08:52.172935 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:52.173367 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.173204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56384)
0928 09:08:52.173266 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.173333 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 09:08:52.173583 23069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
I20260928 09:08:52.174410 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.173988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56388)
0928 09:08:52.174328 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.174363 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
W20260928 09:08:52.174571 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 09:08:52.185902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAWHStC/build/release/bin/kudu
/tmp/dist-test-taskAWHStC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39289
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40881,127.0.98.61:33441,127.0.98.60:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:08:52.221726 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } attempt: 3
I20260928 09:08:52.223814 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } attempt: 1
I20260928 09:08:52.224169 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.224058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56402)
0928 09:08:52.224107 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.224143 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:08:52.224579 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.224456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56406)
0928 09:08:52.224516 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.224552 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 09:08:52.224756 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:52.229192 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.222204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33742)
0928 09:08:52.222282 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.222296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.222348 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.222539 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.222540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.222764 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.222861 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.222864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.223514 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.223515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.224337 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.224339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.224848 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.224849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.224929 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.225327 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.225332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.226650 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.228313 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.228314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.228315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.228479 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.228674 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.228675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.228675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.228724 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.228974 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.228975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.229010 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.229060 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.229130 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 09:08:52.229233 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.222284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.98.61:33441)
0928 09:08:52.222346 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.222348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.222358 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:08:52.222427 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.222428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.222439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.222501 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.222951 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.223428 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.224423 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.224716 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.224790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.226728 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.226729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.226729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.226737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.228232 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.228547 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.228548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.228549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.228609 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.228785 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.228786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.228786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.228913 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.228978 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.229082 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.229154 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 09:08:52.229712 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:52.230051 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.229953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56418)
0928 09:08:52.230000 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.230025 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 09:08:52.230414 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.230327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56430)
0928 09:08:52.230371 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.230396 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:08:52.230520 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
I20260928 09:08:52.261624 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } attempt: 1
I20260928 09:08:52.262120 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.261980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56436)
0928 09:08:52.262049 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.262091 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 09:08:52.262503 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.262416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56438)
0928 09:08:52.262461 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.262485 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:08:52.262632 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
W20260928 09:08:52.270432 23127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:08:52.270660 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:08:52.270696 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:08:52.270736 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:08:52.271971 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:08:52.272037 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:08:52.272061 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:08:52.272091 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:08:52.272120 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:08:52.273761 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40881,127.0.98.61:33441,127.0.98.60:39289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:08:52.274026 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:08:52.274251 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:08:52.279071 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:52.279054 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:08:52.279054 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:08:52.279757 23127 server_base.cc:1034] running on GCE node
I20260928 09:08:52.280017 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:08:52.280303 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:08:52.281474 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790586532281465 us; error 41 us; skew 500 ppm
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:08:52.284000 23127 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 09:08:52.284458 23127 webserver.cc:460] Webserver started at http://127.0.98.60:44089/ using document root <none> and password file <none>
I20260928 09:08:52.284683 23127 fs_manager.cc:359] Metadata directory not provided
I20260928 09:08:52.284758 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:08:52.284930 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:08:52.286093 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data/instance:
uuid: "b392f8c5e27d48eea6aa8a4e00797dc4"
format_stamp: "Formatted at 2026-09-28 09:08:52 on dist-test-slave-smns"
server_key: "e57af16b1ec8943422804c75a3701465"
server_key_iv: "c80e90a2c72e53f0ea2c81b99274fcda"
server_key_version: "encryptionkey@0"
I20260928 09:08:52.286451 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal/instance:
uuid: "b392f8c5e27d48eea6aa8a4e00797dc4"
format_stamp: "Formatted at 2026-09-28 09:08:52 on dist-test-slave-smns"
server_key: "e57af16b1ec8943422804c75a3701465"
server_key_iv: "c80e90a2c72e53f0ea2c81b99274fcda"
server_key_version: "encryptionkey@0"
I20260928 09:08:52.287798 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:08:52.288534 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:08:52.288694 23127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:08:52.288795 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal
uuid: "b392f8c5e27d48eea6aa8a4e00797dc4"
format_stamp: "Formatted at 2026-09-28 09:08:52 on dist-test-slave-smns"
server_key: "e57af16b1ec8943422804c75a3701465"
server_key_iv: "c80e90a2c72e53f0ea2c81b99274fcda"
server_key_version: "encryptionkey@0"
I20260928 09:08:52.288868 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:08:52.292896 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } attempt: 2
I20260928 09:08:52.293272 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.293148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56440)
0928 09:08:52.293206 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.293243 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:08:52.293602 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.293492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56448)
0928 09:08:52.293536 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.293578 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:08:52.293689 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39289: connect: Connection refused (error 111)
I20260928 09:08:52.306465 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:08:52.308615 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:08:52.308804 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:08:52.313036 23127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39289
I20260928 09:08:52.313068 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39289 every 8 connection(s)
I20260928 09:08:52.313484 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:08:52.314473 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:08:52.315999 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:08:52.320693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAWHStC/build/release/bin/kudu as pid 23127
I20260928 09:08:52.320783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAWHStC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790586261566264-392-0/minicluster-data/master-2/wal/instance
I20260928 09:08:52.321035 392 external_mini_cluster.cc:1428] Setting key cf50db4134e2be1e08aa665f895a3e4f
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:08:52.325141 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.316703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.0.98.62:40881)
0928 09:08:52.316775 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.316777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.316952 (+ 175us) server_negotiation.cc:407] Connection header received
0928 09:08:52.317403 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.317404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.317413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.317474 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.318157 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.318631 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.319554 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.319728 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.319778 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.322245 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.322247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.322248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.322257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.323773 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.324158 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.324159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.324159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.324229 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.324463 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.324464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.324464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.324607 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.324673 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.324978 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.325044 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 09:08:52.325428 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.316676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40881 (local address 127.0.0.1:42692)
0928 09:08:52.316827 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.316869 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.317322 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.317605 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.317610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.317867 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.318057 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.318061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.318710 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.318710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.319481 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.319483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.319822 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.319823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.319916 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.320482 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.320490 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.322159 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.323852 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.323857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.323859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.324083 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.324297 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.324297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.324298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.324396 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.324673 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.324677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.324841 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.324954 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.325112 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260928 09:08:52.326107 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:52.330477 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.321471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.0.98.60:39289)
0928 09:08:52.321855 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.321860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.321883 (+ 23us) server_negotiation.cc:407] Connection header received
0928 09:08:52.321905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.321907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.321926 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.322010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.322787 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.323638 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.324619 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.325064 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.325131 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.327579 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.327586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.327586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.327593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.329306 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.329687 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.329690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.329691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.329748 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.329902 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.329903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.329904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.330048 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.330200 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.330284 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.330357 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":100,"threads_started":1}
I20260928 09:08:52.332873 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.326452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.0.98.61:33441)
0928 09:08:52.326504 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.326506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.326514 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:52.326537 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.326538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.326547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.326612 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.326900 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.327453 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.328135 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.328355 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.328430 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.330196 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.330198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.330198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.330205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.331785 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.332122 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.332123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.332124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.332177 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.332377 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.332378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.332379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.332492 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.332564 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.332711 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.332780 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:08:52.332875 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.326369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33441 (local address 127.0.0.1:33756)
0928 09:08:52.326414 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.326425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.326460 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.326627 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.326628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.326760 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.326822 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.326823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.327529 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.327531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.328036 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.328038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.328427 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.328428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.328496 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.328843 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.328847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.330114 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.331862 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.331863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.331863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.332033 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.332250 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.332251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.332251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.332306 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.332557 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.332557 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.332598 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.332686 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.332776 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 09:08:52.333213 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:08:52.337775 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } attempt: 2
I20260928 09:08:52.338675 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.331314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.0.98.62:40881)
0928 09:08:52.331364 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.331366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.331375 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:52.331398 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.331398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.331409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.331475 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.331917 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.332370 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.333162 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.333412 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.333477 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.335730 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.335732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.335732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.335739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.337361 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.337781 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.337783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.337783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.337872 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.338100 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.338101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.338101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.338230 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.338434 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.338476 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.338581 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 09:08:52.341279 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.333449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56472)
0928 09:08:52.333517 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.333526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.333598 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.333745 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.333746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.333901 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.333939 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.333941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.334531 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.334532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.335094 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.335096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.335555 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.335556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.335593 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.335989 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.335995 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.337496 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.339665 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.339666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.339666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.339872 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.340636 (+ 764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.340637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.340637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.340703 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.340974 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.340975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.341012 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.341098 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.341192 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 09:08:52.341727 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.333489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.98.60:39289)
0928 09:08:52.333520 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.333522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.333573 (+ 51us) server_negotiation.cc:407] Connection header received
0928 09:08:52.333652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.333653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.333662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.333717 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.334020 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.334456 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.335180 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.335481 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.335556 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.337607 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.337609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.337609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.337616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.339591 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.340512 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.340513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.340514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.340569 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.340774 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.340775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.340775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.340890 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.340976 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.341131 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.341628 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":419,"server-negotiator.queue_time_us":14}
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:08:52.343298 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: Bootstrap starting.
I20260928 09:08:52.344051 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:52.344347 23195 log.cc:824] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: Log is configured to *not* fsync() on all Append() calls
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:08:52.345095 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: No bootstrap required, opened a new log
I20260928 09:08:52.346259 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.338069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.98.60:39289)
0928 09:08:52.338245 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.338247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.338254 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:08:52.338273 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.338274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.338329 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.338714 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.338772 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.339998 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.340945 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.341263 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.341347 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.343261 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.343262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.343262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.343269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.344923 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.345328 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.345329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.345329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.345393 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.345640 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.345642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.345642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.345804 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.345874 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.346039 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.346112 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260928 09:08:52.346452 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.338032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56478)
0928 09:08:52.338089 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.338100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.338144 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.338428 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.338429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.338607 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.338642 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.338644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.340077 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.340079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.340862 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.340865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.341345 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.341346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.341403 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.341806 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.341811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.343165 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.345014 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.345015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.345016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.345240 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.345478 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.345479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.345479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.345537 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.345867 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.345868 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.345921 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.346013 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.346386 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 09:08:52.347975 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db: Bootstrap starting.
I20260928 09:08:52.348220 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.339733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.0.98.61:33441)
0928 09:08:52.339772 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.339774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.339782 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:08:52.339835 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.339835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.339845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.339905 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.340496 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.341135 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.342268 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.342561 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.342653 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.345348 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.345350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.345350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.345358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.347217 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.347547 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.347548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.347548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.347606 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.347751 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.347751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.347752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.347875 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.348035 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.348074 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.348141 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:08:52.348649 23045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:52.348973 23045 log.cc:824] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:52.349594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:08:52.349799 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db: No bootstrap required, opened a new log
I20260928 09:08:52.350194 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.350359 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b392f8c5e27d48eea6aa8a4e00797dc4, State: Initialized, Role: FOLLOWER
I20260928 09:08:52.350471 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.350792 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:52.350385 23019 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f302ee6a833a457cb754f2cdc62ad5db"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708
I20260928 09:08:52.352097 23045 raft_consensus.cc:348] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.352234 23045 raft_consensus.cc:729] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f302ee6a833a457cb754f2cdc62ad5db, State: Initialized, Role: FOLLOWER
I20260928 09:08:52.352329 23045 consensus_queue.cc:260] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.350751 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.352605 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:52.352687 23045 sys_catalog.cc:564] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:52.352734 23019 raft_consensus.cc:482] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
W20260928 09:08:52.352753 23214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:52.352799 23214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:52.352800 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:52.353840 23019 raft_consensus.cc:504] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.354447 23169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f302ee6a833a457cb754f2cdc62ad5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4"
I20260928 09:08:52.354557 23169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:52.354120 23019 leader_election.cc:288] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [CANDIDATE]: Term 1 election: Requested vote from peers 532524efa57f42d5ad19bd03e77e28cf (127.0.98.61:33441), b392f8c5e27d48eea6aa8a4e00797dc4 (127.0.98.60:39289)
I20260928 09:08:52.353052 23127 master_runner.cc:186] Error getting master registration for 127.0.98.62:40881: OK, instance_id { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" instance_seqno: 1790586531994331 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:08:52.355399 23127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:08:52.355458 23169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f302ee6a833a457cb754f2cdc62ad5db in term 1.
I20260928 09:08:52.355500 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.355571 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:52.356081 22994 leader_election.cc:302] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b392f8c5e27d48eea6aa8a4e00797dc4, f302ee6a833a457cb754f2cdc62ad5db; no voters:
W20260928 09:08:52.356534 23227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:52.356645 23227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:52.357025 23215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:08:52.357246 23215 raft_consensus.cc:686] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [term 1 LEADER]: Becoming Leader. State: Replica: f302ee6a833a457cb754f2cdc62ad5db, State: Running, Role: LEADER
I20260928 09:08:52.357370 23215 consensus_queue.cc:237] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.357868 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f302ee6a833a457cb754f2cdc62ad5db. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.358014 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:52.355289 22977 master_runner.cc:186] Error getting master registration for 127.0.98.61:33441: OK, instance_id { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" instance_seqno: 1790586532149689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:08:52.358296 22977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:08:52.366000 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.356772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.98.61:33441)
0928 09:08:52.356829 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.356831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.357097 (+ 266us) server_negotiation.cc:407] Connection header received
0928 09:08:52.357207 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.357208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.357221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.357251 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.358115 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.358828 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.359848 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.360161 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.360233 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.362477 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.362478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.362478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.362486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.364618 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.364993 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.364994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.364994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.365066 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.365267 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.365268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.365268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.365400 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.365508 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.365823 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.365907 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:08:52.369762 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.356608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.0.98.60:39289)
0928 09:08:52.358371 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.358373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.359939 (+ 1566us) server_negotiation.cc:407] Connection header received
0928 09:08:52.360038 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.360040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.360052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.360082 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.360817 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.362048 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.363383 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.363744 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.363790 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.366386 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.366387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.366387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.366395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.368091 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.368661 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.368662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.368662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.368719 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.369024 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.369025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.369026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.369165 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.369251 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.369586 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.369674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1741}
I20260928 09:08:52.370271 23238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:08:52.370493 23238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:08:52.372495 23169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [term 1 FOLLOWER]: Refusing update from remote peer f302ee6a833a457cb754f2cdc62ad5db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:52.372936 23236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:52.373275 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.356411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.0.98.62:40881)
0928 09:08:52.356447 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.356449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.361706 (+ 5257us) server_negotiation.cc:407] Connection header received
0928 09:08:52.361822 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.361824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.361836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.361867 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.365889 (+ 4022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.366531 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.367954 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.368237 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.368314 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.369837 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.369839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.369840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.369849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.371599 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.372028 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.372030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.372030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.372086 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.372291 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.372292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.372293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.372441 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.372538 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.372883 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.373025 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:08:52.374518 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f302ee6a833a457cb754f2cdc62ad5db. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.374614 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:52.384858 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } attempt: 3
I20260928 09:08:52.386272 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.386363 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:52.386404 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.386483 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:52.386588 23239 mvcc.cc:205] Tried to move back new op lower bound from 7334242436594774016 to 7334242436536778752. Current Snapshot: MvccSnapshot[applied={T|T < 7334242436594774016}]
I20260928 09:08:52.386742 23240 mvcc.cc:205] Tried to move back new op lower bound from 7334242436594774016 to 7334242436536778752. Current Snapshot: MvccSnapshot[applied={T|T < 7334242436594774016}]
Sep 28 09:08:52 dist-test-slave-smns krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:08:52.393265 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.385181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39289 (local address 127.0.0.1:56498)
0928 09:08:52.385235 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:08:52.385248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:08:52.385289 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:08:52.385785 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:08:52.385786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:08:52.385969 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:08:52.386018 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.386021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.386947 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.386948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.387785 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:08:52.387792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.388272 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.388273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:08:52.388331 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.388787 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:08:52.388792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:08:52.390262 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:08:52.392120 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.392121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.392121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.392364 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.392605 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:08:52.392606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:08:52.392606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:08:52.392670 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:08:52.392951 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:08:52.392952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:08:52.393009 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 09:08:52.393074 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 09:08:52.393164 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:08:52.393285 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:52.385498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.98.60:39289)
0928 09:08:52.385627 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.385629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.385636 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:08:52.385656 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.385657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.385667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.385731 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.386105 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.386805 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.387880 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:52.388189 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:52.388260 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:52.390352 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:08:52.390354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:08:52.390354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:08:52.390363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:08:52.392012 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.392460 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.392461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.392461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.392539 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:08:52.392738 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:08:52.392739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:08:52.392740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:08:52.392865 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:08:52.392944 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:52.393122 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 09:08:52.393197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 09:08:52.395298 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: Bootstrap starting.
I20260928 09:08:52.396013 23120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: Neither blocks nor log segments found. Creating new log.
I20260928 09:08:52.396310 23120 log.cc:824] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: Log is configured to *not* fsync() on all Append() calls
I20260928 09:08:52.397388 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: No bootstrap required, opened a new log
I20260928 09:08:52.397948 23238 catalog_manager.cc:1347] Generated new cluster ID: d1fb327ceb894ec1be9eb9cb675a5e4f
I20260928 09:08:52.398000 23238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:08:52.399449 23120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.399590 23120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532524efa57f42d5ad19bd03e77e28cf, State: Initialized, Role: FOLLOWER
I20260928 09:08:52.399672 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } }
I20260928 09:08:52.399984 23120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:08:52.400094 23094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f302ee6a833a457cb754f2cdc62ad5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "532524efa57f42d5ad19bd03e77e28cf"
I20260928 09:08:52.400200 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:08:52.400959 23094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f302ee6a833a457cb754f2cdc62ad5db in term 1.
I20260928 09:08:52.401365 23093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [term 1 FOLLOWER]: Refusing update from remote peer f302ee6a833a457cb754f2cdc62ad5db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:08:52.401613 23236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:08:52.404791 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.404891 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:52.404937 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f302ee6a833a457cb754f2cdc62ad5db. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.404983 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:08:52.405010 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.405053 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf [sys.catalog]: This master's current role is: FOLLOWER
W20260928 09:08:52.406404 23256 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:08:52.406466 23256 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:08:52.409783 23258 mvcc.cc:205] Tried to move back new op lower bound from 7334242436594774016 to 7334242436536778752. Current Snapshot: MvccSnapshot[applied={T|T < 7334242436594774016}]
I20260928 09:08:52.410789 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f302ee6a833a457cb754f2cdc62ad5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f302ee6a833a457cb754f2cdc62ad5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } peers { permanent_uuid: "532524efa57f42d5ad19bd03e77e28cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33441 } } peers { permanent_uuid: "b392f8c5e27d48eea6aa8a4e00797dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39289 } } }
I20260928 09:08:52.410887 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db [sys.catalog]: This master's current role is: LEADER
I20260928 09:08:52.412308 23238 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:08:52.412999 23238 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:08:52.418193 23238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f302ee6a833a457cb754f2cdc62ad5db: Generated new TSK 0
I20260928 09:08:52.418356 23238 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:08:52.438210 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.436686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.98.61:33441)
0928 09:08:52.436747 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.436749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.437441 (+ 692us) server_negotiation.cc:407] Connection header received
0928 09:08:52.437587 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.437588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.437599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.437632 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.438135 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33786: BlockingRecv error: recv got EOF from 127.0.0.1:33786 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 09:08:52.438956 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.437159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.98.60:39289)
0928 09:08:52.437202 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.437204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.438448 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 09:08:52.438543 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.438544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.438554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.438583 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.438894 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56514: BlockingRecv error: recv got EOF from 127.0.0.1:56514 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 09:08:52.442559 23048 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.441816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.0.98.62:40881)
0928 09:08:52.441861 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.441863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.441987 (+ 124us) server_negotiation.cc:407] Connection header received
0928 09:08:52.442088 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.442089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.442099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.442128 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.442491 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42722: BlockingRecv error: recv got EOF from 127.0.0.1:42722 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 09:08:52.444988 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.444144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.0.98.61:33441)
0928 09:08:52.444194 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.444196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.444432 (+ 236us) server_negotiation.cc:407] Connection header received
0928 09:08:52.444545 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.444546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.444569 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.444600 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.444936 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33794: BlockingRecv error: recv got EOF from 127.0.0.1:33794 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 09:08:52.447891 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.447114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.0.98.61:33441)
0928 09:08:52.447146 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.447148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.447397 (+ 249us) server_negotiation.cc:407] Connection header received
0928 09:08:52.447490 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.447491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.447509 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.447538 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.447844 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33800: BlockingRecv error: recv got EOF from 127.0.0.1:33800 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 09:08:52.451786 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.450983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.0.98.61:33441)
0928 09:08:52.451022 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.451024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.451277 (+ 253us) server_negotiation.cc:407] Connection header received
0928 09:08:52.451386 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.451387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.451397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.451425 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.451743 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33816: BlockingRecv error: recv got EOF from 127.0.0.1:33816 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 09:08:52.457612 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.456822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.0.98.61:33441)
0928 09:08:52.456857 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.456858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.457091 (+ 233us) server_negotiation.cc:407] Connection header received
0928 09:08:52.457181 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.457182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.457192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.457221 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.457521 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33820: BlockingRecv error: recv got EOF from 127.0.0.1:33820 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 09:08:52.467599 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.466729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.0.98.61:33441)
0928 09:08:52.466768 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.466770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.467026 (+ 256us) server_negotiation.cc:407] Connection header received
0928 09:08:52.467129 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.467131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.467141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.467171 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.467531 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33836: BlockingRecv error: recv got EOF from 127.0.0.1:33836 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 09:08:52.486048 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.484967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.0.98.61:33441)
0928 09:08:52.485015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.485017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.485355 (+ 338us) server_negotiation.cc:407] Connection header received
0928 09:08:52.485468 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.485469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.485482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.485514 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.485984 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33840: BlockingRecv error: recv got EOF from 127.0.0.1:33840 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:08:52.520370 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.519385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.0.98.61:33441)
0928 09:08:52.519432 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.519435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.519743 (+ 308us) server_negotiation.cc:407] Connection header received
0928 09:08:52.519865 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.519866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.519879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.519911 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.520308 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33856: BlockingRecv error: recv got EOF from 127.0.0.1:33856 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:08:52.586625 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.585799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.0.98.61:33441)
0928 09:08:52.585871 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.585874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.585996 (+ 122us) server_negotiation.cc:407] Connection header received
0928 09:08:52.586058 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.586059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.586069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.586123 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.586535 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33864: BlockingRecv error: recv got EOF from 127.0.0.1:33864 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 09:08:52.716403 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.715691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.0.98.61:33441)
0928 09:08:52.715757 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.715759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.715878 (+ 119us) server_negotiation.cc:407] Connection header received
0928 09:08:52.715951 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.715952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.715964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.716018 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.716336 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33870: BlockingRecv error: recv got EOF from 127.0.0.1:33870 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 09:08:52.974574 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:08:52.973682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.0.98.61:33441)
0928 09:08:52.973756 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:52.973759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:52.973850 (+ 91us) server_negotiation.cc:407] Connection header received
0928 09:08:52.973966 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:52.973967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:52.973980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:52.974054 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:08:52.974492 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33884: BlockingRecv error: recv got EOF from 127.0.0.1:33884 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:08:53.353314 23214 catalog_manager.cc:1259] Loaded cluster ID: d1fb327ceb894ec1be9eb9cb675a5e4f
I20260928 09:08:53.353389 23214 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: loading cluster ID for follower catalog manager: success
I20260928 09:08:53.354328 23214 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: acquiring CA information for follower catalog manager: success
I20260928 09:08:53.354630 23214 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b392f8c5e27d48eea6aa8a4e00797dc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:08:53.406992 23256 catalog_manager.cc:1259] Loaded cluster ID: d1fb327ceb894ec1be9eb9cb675a5e4f
I20260928 09:08:53.407078 23256 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: loading cluster ID for follower catalog manager: success
I20260928 09:08:53.408062 23256 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: acquiring CA information for follower catalog manager: success
I20260928 09:08:53.408434 23256 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 532524efa57f42d5ad19bd03e77e28cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:08:53.490729 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:53.487872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.0.98.61:33441)
0928 09:08:53.488007 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:53.488010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:53.488090 (+ 80us) server_negotiation.cc:407] Connection header received
0928 09:08:53.488161 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:53.488162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:53.488238 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:53.488330 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:53.488524 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.489043 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.489859 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.490082 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.490163 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:53.490349 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:53.490474 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:53.490564 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:53.490605 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 09:08:53.490628 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260928 09:08:53.495286 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:53.492758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.0.98.60:39289)
0928 09:08:53.492877 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:53.492879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:53.492888 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:08:53.492982 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:53.492983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:53.493052 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:53.493138 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:53.493293 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.493851 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.494609 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.494802 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.494855 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:53.494999 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:53.495100 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:53.495158 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:53.495204 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 09:08:53.495220 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
I20260928 09:08:53.501014 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:53.497688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.98.60:39289)
0928 09:08:53.497891 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:53.497903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:53.498168 (+ 265us) server_negotiation.cc:407] Connection header received
0928 09:08:53.498281 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:53.498283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:53.498294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:53.498323 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:53.498497 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.499018 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.500191 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.500471 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.500532 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:53.500751 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:53.500782 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:53.500807 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:53.500916 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 09:08:53.500938 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 09:08:53.502087 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:53.497985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.0.98.61:33441)
0928 09:08:53.498023 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:53.498025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:53.498250 (+ 225us) server_negotiation.cc:407] Connection header received
0928 09:08:53.498312 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:53.498313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:53.498329 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:53.498400 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:53.498501 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.499378 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.501163 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.501434 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.501468 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:53.501789 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:53.501825 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:53.501854 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:53.501971 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 09:08:53.501994 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:08:53.504034 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:08:53.497594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.98.62:40881)
0928 09:08:53.500231 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0928 09:08:53.500234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:08:53.500247 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:08:53.500315 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:08:53.500316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:08:53.500375 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:08:53.500674 (+ 299us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:08:53.500727 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.501663 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.502919 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:08:53.503302 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:08:53.503386 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:08:53.503608 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:08:53.503722 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:08:53.503783 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:08:53.503887 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:08:53.503906 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
I20260928 09:08:53.528685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 22977
I20260928 09:08:53.534157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 23052
I20260928 09:08:53.539187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAWHStC/build/release/bin/kudu with pid 23127
2026-09-28T09:08:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2852 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5692 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271979 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST